Mercurial > hg > CbC > CbC_llvm

annotate docs/TestSuiteMakefileGuide.rst @ 131:f476a9ba4795

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
http://llvm.org/svn/llvm-project/compiler-rt/trunk compiler-rt
author mir3636
date Fri, 16 Feb 2018 21:02:11 +0900
parents 1172e4bd9c6f
children c2174574ed3a
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
120
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
1 =====================
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
2 LLVM test-suite Guide
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
3 =====================
0
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
4
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
5 .. contents::
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
6 :local:
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
7
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
8 Overview
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
9 ========
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
10
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
11 This document describes the features of the Makefile-based LLVM
120
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
12 test-suite as well as the cmake based replacement. This way of interacting
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
13 with the test-suite is deprecated in favor of running the test-suite using LNT,
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
14 but may continue to prove useful for some users. See the Testing
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
15 Guide's :ref:`test-suite Quickstart <test-suite-quickstart>` section for more
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
16 information.
0
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
17
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
18 Test suite Structure
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
19 ====================
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
20
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
21 The ``test-suite`` module contains a number of programs that can be
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
22 compiled with LLVM and executed. These programs are compiled using the
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
23 native compiler and various LLVM backends. The output from the program
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
24 compiled with the native compiler is assumed correct; the results from
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
25 the other programs are compared to the native program output and pass if
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
26 they match.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
27
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
28 When executing tests, it is usually a good idea to start out with a
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
29 subset of the available tests or programs. This makes test run times
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
30 smaller at first and later on this is useful to investigate individual
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
31 test failures. To run some test only on a subset of programs, simply
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
32 change directory to the programs you want tested and run ``gmake``
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
33 there. Alternatively, you can run a different test using the ``TEST``
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
34 variable to change what tests or run on the selected programs (see below
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
35 for more info).
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
36
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
37 In addition for testing correctness, the ``test-suite`` directory also
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
38 performs timing tests of various LLVM optimizations. It also records
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
39 compilation times for the compilers and the JIT. This information can be
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
40 used to compare the effectiveness of LLVM's optimizations and code
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
41 generation.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
42
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
43 ``test-suite`` tests are divided into three types of tests: MultiSource,
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
44 SingleSource, and External.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
45
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
46 - ``test-suite/SingleSource``
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
47
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
48 The SingleSource directory contains test programs that are only a
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
49 single source file in size. These are usually small benchmark
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
50 programs or small programs that calculate a particular value. Several
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
51 such programs are grouped together in each directory.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
52
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
53 - ``test-suite/MultiSource``
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
54
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
55 The MultiSource directory contains subdirectories which contain
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
56 entire programs with multiple source files. Large benchmarks and
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
57 whole applications go here.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
58
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
59 - ``test-suite/External``
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
60
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
61 The External directory contains Makefiles for building code that is
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
62 external to (i.e., not distributed with) LLVM. The most prominent
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
63 members of this directory are the SPEC 95 and SPEC 2000 benchmark
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
64 suites. The ``External`` directory does not contain these actual
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
65 tests, but only the Makefiles that know how to properly compile these
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
66 programs from somewhere else. The presence and location of these
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
67 external programs is configured by the test-suite ``configure``
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
68 script.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
69
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
70 Each tree is then subdivided into several categories, including
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
71 applications, benchmarks, regression tests, code that is strange
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
72 grammatically, etc. These organizations should be relatively self
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
73 explanatory.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
74
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
75 Some tests are known to fail. Some are bugs that we have not fixed yet;
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
76 others are features that we haven't added yet (or may never add). In the
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
77 regression tests, the result for such tests will be XFAIL (eXpected
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
78 FAILure). In this way, you can tell the difference between an expected
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
79 and unexpected failure.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
80
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
81 The tests in the test suite have no such feature at this time. If the
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
82 test passes, only warnings and other miscellaneous output will be
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
83 generated. If a test fails, a large <program> FAILED message will be
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
84 displayed. This will help you separate benign warnings from actual test
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
85 failures.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
86
120
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
87 Running the test suite via CMake
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
88 ================================
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
89
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
90 To run the test suite, you need to use the following steps:
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
91
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
92 #. The test suite uses the lit test runner to run the test-suite,
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
93 you need to have lit installed first. Check out LLVM and install lit:
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
94
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
95 .. code-block:: bash
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
96
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
97 % svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
98 % cd llvm/utils/lit
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
99 % sudo python setup.py install # Or without sudo, install in virtual-env.
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
100 running install
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
101 running bdist_egg
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
102 running egg_info
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
103 writing lit.egg-info/PKG-INFO
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
104 ...
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
105 % lit --version
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
106 lit 0.5.0dev
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
107
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
108 #. Check out the ``test-suite`` module with:
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
109
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
110 .. code-block:: bash
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
111
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
112 % svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
113
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
114 #. Use CMake to configure the test suite in a new directory. You cannot build
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
115 the test suite in the source tree.
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
116
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
117 .. code-block:: bash
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
118
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
119 % mkdir test-suite-build
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
120 % cd test-suite-build
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
121 % cmake ../test-suite
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
122
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
123 #. Build the benchmarks, using the makefiles CMake generated.
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
124
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
125 .. code-block:: bash
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
126
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
127 % make
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
128 Scanning dependencies of target timeit-target
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
129 [ 0%] Building C object tools/CMakeFiles/timeit-target.dir/timeit.c.o
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
130 [ 0%] Linking C executable timeit-target
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
131 [ 0%] Built target timeit-target
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
132 Scanning dependencies of target fpcmp-host
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
133 [ 0%] [TEST_SUITE_HOST_CC] Building host executable fpcmp
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
134 [ 0%] Built target fpcmp-host
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
135 Scanning dependencies of target timeit-host
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
136 [ 0%] [TEST_SUITE_HOST_CC] Building host executable timeit
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
137 [ 0%] Built target timeit-host
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
138
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
139
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
140 #. Run the tests with lit:
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
141
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
142 .. code-block:: bash
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
143
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
144 % lit -v -j 1 . -o results.json
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
145 -- Testing: 474 tests, 1 threads --
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
146 PASS: test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test (1 of 474)
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
147 ********** TEST 'test-suite :: MultiSource/Applications/ALAC/decode/alacconvert-decode.test' RESULTS **********
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
148 compile_time: 0.2192
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
149 exec_time: 0.0462
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
150 hash: "59620e187c6ac38b36382685ccd2b63b"
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
151 size: 83348
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
152 **********
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
153 PASS: test-suite :: MultiSource/Applications/ALAC/encode/alacconvert-encode.test (2 of 474)
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
154
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
155
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
156 Running the test suite via Makefiles (deprecated)
1172e4bd9c6f update 4.0.0
mir3636
parents: 0
diff changeset
157 =================================================
0
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
158
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
159 First, all tests are executed within the LLVM object directory tree.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
160 They *are not* executed inside of the LLVM source tree. This is because
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
161 the test suite creates temporary files during execution.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
162
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
163 To run the test suite, you need to use the following steps:
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
164
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
165 #. ``cd`` into the ``llvm/projects`` directory in your source tree.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
166 #. Check out the ``test-suite`` module with:
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
167
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
168 .. code-block:: bash
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
169
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
170 % svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
171
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
172 This will get the test suite into ``llvm/projects/test-suite``.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
173
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
174 #. Configure and build ``llvm``.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
175
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
176 #. Configure and build ``llvm-gcc``.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
177
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
178 #. Install ``llvm-gcc`` somewhere.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
179
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
180 #. *Re-configure* ``llvm`` from the top level of each build tree (LLVM
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
181 object directory tree) in which you want to run the test suite, just
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
182 as you do before building LLVM.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
183
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
184 During the *re-configuration*, you must either: (1) have ``llvm-gcc``
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
185 you just built in your path, or (2) specify the directory where your
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
186 just-built ``llvm-gcc`` is installed using
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
187 ``--with-llvmgccdir=$LLVM_GCC_DIR``.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
188
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
189 You must also tell the configure machinery that the test suite is
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
190 available so it can be configured for your build tree:
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
191
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
192 .. code-block:: bash
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
193
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
194 % cd $LLVM_OBJ_ROOT ; $LLVM_SRC_ROOT/configure [--with-llvmgccdir=$LLVM_GCC_DIR]
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
195
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
196 [Remember that ``$LLVM_GCC_DIR`` is the directory where you
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
197 *installed* llvm-gcc, not its src or obj directory.]
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
198
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
199 #. You can now run the test suite from your build tree as follows:
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
200
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
201 .. code-block:: bash
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
202
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
203 % cd $LLVM_OBJ_ROOT/projects/test-suite
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
204 % make
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
205
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
206 Note that the second and third steps only need to be done once. After
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
207 you have the suite checked out and configured, you don't need to do it
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
208 again (unless the test code or configure script changes).
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
209
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
210 Configuring External Tests
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
211 --------------------------
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
212
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
213 In order to run the External tests in the ``test-suite`` module, you
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
214 must specify *--with-externals*. This must be done during the
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
215 *re-configuration* step (see above), and the ``llvm`` re-configuration
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
216 must recognize the previously-built ``llvm-gcc``. If any of these is
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
217 missing or neglected, the External tests won't work.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
218
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
219 * *--with-externals*
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
220
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
221 * *--with-externals=<directory>*
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
222
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
223 This tells LLVM where to find any external tests. They are expected to
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
224 be in specifically named subdirectories of <``directory``>. If
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
225 ``directory`` is left unspecified, ``configure`` uses the default value
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
226 ``/home/vadve/shared/benchmarks/speccpu2000/benchspec``. Subdirectory
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
227 names known to LLVM include:
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
228
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
229 * spec95
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
230
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
231 * speccpu2000
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
232
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
233 * speccpu2006
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
234
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
235 * povray31
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
236
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
237 Others are added from time to time, and can be determined from
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
238 ``configure``.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
239
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
240 Running different tests
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
241 -----------------------
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
242
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
243 In addition to the regular "whole program" tests, the ``test-suite``
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
244 module also provides a mechanism for compiling the programs in different
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
245 ways. If the variable TEST is defined on the ``gmake`` command line, the
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
246 test system will include a Makefile named
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
247 ``TEST.<value of TEST variable>.Makefile``. This Makefile can modify
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
248 build rules to yield different results.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
249
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
250 For example, the LLVM nightly tester uses ``TEST.nightly.Makefile`` to
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
251 create the nightly test reports. To run the nightly tests, run
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
252 ``gmake TEST=nightly``.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
253
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
254 There are several TEST Makefiles available in the tree. Some of them are
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
255 designed for internal LLVM research and will not work outside of the
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
256 LLVM research group. They may still be valuable, however, as a guide to
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
257 writing your own TEST Makefile for any optimization or analysis passes
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
258 that you develop with LLVM.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
259
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
260 Generating test output
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
261 ----------------------
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
262
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
263 There are a number of ways to run the tests and generate output. The
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
264 most simple one is simply running ``gmake`` with no arguments. This will
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
265 compile and run all programs in the tree using a number of different
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
266 methods and compare results. Any failures are reported in the output,
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
267 but are likely drowned in the other output. Passes are not reported
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
268 explicitly.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
269
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
270 Somewhat better is running ``gmake TEST=sometest test``, which runs the
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
271 specified test and usually adds per-program summaries to the output
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
272 (depending on which sometest you use). For example, the ``nightly`` test
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
273 explicitly outputs TEST-PASS or TEST-FAIL for every test after each
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
274 program. Though these lines are still drowned in the output, it's easy
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
275 to grep the output logs in the Output directories.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
276
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
277 Even better are the ``report`` and ``report.format`` targets (where
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
278 ``format`` is one of ``html``, ``csv``, ``text`` or ``graphs``). The
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
279 exact contents of the report are dependent on which ``TEST`` you are
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
280 running, but the text results are always shown at the end of the run and
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
281 the results are always stored in the ``report.<type>.format`` file (when
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
282 running with ``TEST=<type>``). The ``report`` also generate a file
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
283 called ``report.<type>.raw.out`` containing the output of the entire
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
284 test run.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
285
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
286 Writing custom tests for the test suite
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
287 ---------------------------------------
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
288
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
289 Assuming you can run the test suite, (e.g.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
290 "``gmake TEST=nightly report``" should work), it is really easy to run
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
291 optimizations or code generator components against every program in the
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
292 tree, collecting statistics or running custom checks for correctness. At
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
293 base, this is how the nightly tester works, it's just one example of a
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
294 general framework.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
295
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
296 Lets say that you have an LLVM optimization pass, and you want to see
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
297 how many times it triggers. First thing you should do is add an LLVM
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
298 `statistic <ProgrammersManual.html#Statistic>`_ to your pass, which will
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
299 tally counts of things you care about.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
300
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
301 Following this, you can set up a test and a report that collects these
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
302 and formats them for easy viewing. This consists of two files, a
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
303 "``test-suite/TEST.XXX.Makefile``" fragment (where XXX is the name of
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
304 your test) and a "``test-suite/TEST.XXX.report``" file that indicates
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
305 how to format the output into a table. There are many example reports of
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
306 various levels of sophistication included with the test suite, and the
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
307 framework is very general.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
308
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
309 If you are interested in testing an optimization pass, check out the
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
310 "libcalls" test as an example. It can be run like this:
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
311
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
312 .. code-block:: bash
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
313
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
314 % cd llvm/projects/test-suite/MultiSource/Benchmarks # or some other level
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
315 % make TEST=libcalls report
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
316
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
317 This will do a bunch of stuff, then eventually print a table like this:
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
318
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
319 ::
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
320
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
321 Name | total | #exit |
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
322 ...
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
323 FreeBench/analyzer/analyzer | 51 | 6 |
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
324 FreeBench/fourinarow/fourinarow | 1 | 1 |
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
325 FreeBench/neural/neural | 19 | 9 |
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
326 FreeBench/pifft/pifft | 5 | 3 |
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
327 MallocBench/cfrac/cfrac | 1 | * |
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
328 MallocBench/espresso/espresso | 52 | 12 |
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
329 MallocBench/gs/gs | 4 | * |
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
330 Prolangs-C/TimberWolfMC/timberwolfmc | 302 | * |
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
331 Prolangs-C/agrep/agrep | 33 | 12 |
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
332 Prolangs-C/allroots/allroots | * | * |
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
333 Prolangs-C/assembler/assembler | 47 | * |
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
334 Prolangs-C/bison/mybison | 74 | * |
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
335 ...
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
336
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
337 This basically is grepping the -stats output and displaying it in a
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
338 table. You can also use the "TEST=libcalls report.html" target to get
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
339 the table in HTML form, similarly for report.csv and report.tex.
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
340
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
341 The source for this is in ``test-suite/TEST.libcalls.*``. The format is
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
342 pretty simple: the Makefile indicates how to run the test (in this case,
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
343 "``opt -simplify-libcalls -stats``"), and the report contains one line
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
344 for each column of the output. The first value is the header for the
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
345 column and the second is the regex to grep the output of the command
95c75e76d11b LLVM 3.4
Kaito Tokumori <e105711@ie.u-ryukyu.ac.jp>
parents:
diff changeset
346 for. There are lots of example reports that can do fancy stuff.