|
150
|
1 ==============
|
|
|
2 Testing libc++
|
|
|
3 ==============
|
|
|
4
|
|
|
5 .. contents::
|
|
|
6 :local:
|
|
|
7
|
|
|
8 Getting Started
|
|
|
9 ===============
|
|
|
10
|
|
173
|
11 libc++ uses LIT to configure and run its tests.
|
|
150
|
12
|
|
173
|
13 The primary way to run the libc++ tests is by using ``make check-cxx``.
|
|
150
|
14
|
|
|
15 However since libc++ can be used in any number of possible
|
|
|
16 configurations it is important to customize the way LIT builds and runs
|
|
|
17 the tests. This guide provides information on how to use LIT directly to
|
|
|
18 test libc++.
|
|
|
19
|
|
|
20 Please see the `Lit Command Guide`_ for more information about LIT.
|
|
|
21
|
|
173
|
22 .. _LIT Command Guide: https://llvm.org/docs/CommandGuide/lit.html
|
|
150
|
23
|
|
173
|
24 Usage
|
|
|
25 -----
|
|
150
|
26
|
|
173
|
27 After building libc++, you can run parts of the libc++ test suite by simply
|
|
|
28 running ``llvm-lit`` on a specified test or directory. If you're unsure
|
|
|
29 whether the required libraries have been built, you can use the
|
|
|
30 `check-cxx-deps` target. For example:
|
|
150
|
31
|
|
|
32 .. code-block:: bash
|
|
|
33
|
|
173
|
34 $ cd <monorepo-root>
|
|
|
35 $ make -C <build> check-cxx-deps # If you want to make sure the targets get rebuilt
|
|
|
36 $ <build>/bin/llvm-lit -sv libcxx/test/std/re # Run all of the std::regex tests
|
|
|
37 $ <build>/bin/llvm-lit -sv libcxx/test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test
|
|
|
38 $ <build>/bin/llvm-lit -sv libcxx/test/std/atomics libcxx/test/std/threads # Test std::thread and std::atomic
|
|
150
|
39
|
|
|
40 Sometimes you'll want to change the way LIT is running the tests. Custom options
|
|
|
41 can be specified using the `--param=<name>=<val>` flag. The most common option
|
|
|
42 you'll want to change is the standard dialect (ie -std=c++XX). By default the
|
|
|
43 test suite will select the newest C++ dialect supported by the compiler and use
|
|
|
44 that. However if you want to manually specify the option like so:
|
|
|
45
|
|
|
46 .. code-block:: bash
|
|
|
47
|
|
173
|
48 $ <build>/bin/llvm-lit -sv libcxx/test/std/containers # Run the tests with the newest -std
|
|
|
49 $ <build>/bin/llvm-lit -sv libcxx/test/std/containers --param=std=c++03 # Run the tests in C++03
|
|
150
|
50
|
|
|
51 Occasionally you'll want to add extra compile or link flags when testing.
|
|
|
52 You can do this as follows:
|
|
|
53
|
|
|
54 .. code-block:: bash
|
|
|
55
|
|
173
|
56 $ <build>/bin/llvm-lit -sv libcxx/test --param=compile_flags='-Wcustom-warning'
|
|
|
57 $ <build>/bin/llvm-lit -sv libcxx/test --param=link_flags='-L/custom/library/path'
|
|
150
|
58
|
|
|
59 Some other common examples include:
|
|
|
60
|
|
|
61 .. code-block:: bash
|
|
|
62
|
|
|
63 # Specify a custom compiler.
|
|
173
|
64 $ <build>/bin/llvm-lit -sv libcxx/test/std --param=cxx_under_test=/opt/bin/g++
|
|
150
|
65
|
|
|
66 # Enable warnings in the test suite
|
|
173
|
67 $ <build>/bin/llvm-lit -sv libcxx/test --param=enable_warnings=true
|
|
150
|
68
|
|
|
69 # Use UBSAN when running the tests.
|
|
173
|
70 $ <build>/bin/llvm-lit -sv libcxx/test --param=use_sanitizer=Undefined
|
|
|
71
|
|
|
72 Using a custom site configuration
|
|
|
73 ---------------------------------
|
|
150
|
74
|
|
173
|
75 By default, the libc++ test suite will use a site configuration that matches
|
|
|
76 the current CMake configuration. It does so by generating a ``lit.site.cfg``
|
|
|
77 file in the build directory from the ``libcxx/test/lit.site.cfg.in`` template,
|
|
|
78 and pointing ``llvm-lit`` (which is a wrapper around ``llvm/utils/lit/lit.py``)
|
|
|
79 to that file. So when you're running ``<build>/bin/llvm-lit``, the generated
|
|
|
80 ``lit.site.cfg`` file is always loaded first, followed by the actual config in
|
|
|
81 ``libcxx/test/lit.cfg``. However, it is sometimes desirable to use a custom
|
|
|
82 site configuration. To do that, you can use ``--param=libcxx_site_config`` or
|
|
|
83 the ``LIBCXX_SITE_CONFIG`` environment variable to point to the right site
|
|
|
84 configuration file. However, you must stop using ``llvm-lit``, or else the
|
|
|
85 generated ``lit.site.cfg`` will still be preferred:
|
|
|
86
|
|
|
87 .. code-block:: bash
|
|
|
88
|
|
|
89 $ LIBCXX_SITE_CONFIG=path/to/your/site/configuration llvm/utils/lit/lit.py -sv ...
|
|
|
90
|
|
|
91 $ llvm/utils/lit/lit.py -sv ... --param=libcxx_site_config=path/to/your/site/configuration
|
|
|
92
|
|
|
93 In both of these cases, your custom site configuration should set up the
|
|
|
94 ``config`` object in a way that is compatible with what libc++'s ``config.py``
|
|
|
95 module expects.
|
|
150
|
96
|
|
|
97 LIT Options
|
|
|
98 ===========
|
|
|
99
|
|
|
100 :program:`lit` [*options*...] [*filenames*...]
|
|
|
101
|
|
|
102 Command Line Options
|
|
|
103 --------------------
|
|
|
104
|
|
173
|
105 To use these options you pass them on the LIT command line as ``--param NAME``
|
|
|
106 or ``--param NAME=VALUE``. Some options have default values specified during
|
|
|
107 CMake's configuration. Passing the option on the command line will override the
|
|
|
108 default.
|
|
150
|
109
|
|
|
110 .. program:: lit
|
|
|
111
|
|
|
112 .. option:: cxx_under_test=<path/to/compiler>
|
|
|
113
|
|
|
114 Specify the compiler used to build the tests.
|
|
|
115
|
|
|
116 .. option:: cxx_stdlib_under_test=<stdlib name>
|
|
|
117
|
|
|
118 **Values**: libc++, libstdc++
|
|
|
119
|
|
|
120 Specify the C++ standard library being tested. Unless otherwise specified
|
|
|
121 libc++ is used. This option is intended to allow running the libc++ test
|
|
|
122 suite against other standard library implementations.
|
|
|
123
|
|
|
124 .. option:: std=<standard version>
|
|
|
125
|
|
|
126 **Values**: c++98, c++03, c++11, c++14, c++17, c++2a
|
|
|
127
|
|
|
128 Change the standard version used when building the tests.
|
|
|
129
|
|
|
130 .. option:: libcxx_site_config=<path/to/lit.site.cfg>
|
|
|
131
|
|
|
132 Specify the site configuration to use when running the tests. This option
|
|
|
133 overrides the environment variable LIBCXX_SITE_CONFIG.
|
|
|
134
|
|
|
135 .. option:: cxx_headers=<path/to/headers>
|
|
|
136
|
|
|
137 Specify the c++ standard library headers that are tested. By default the
|
|
|
138 headers in the source tree are used.
|
|
|
139
|
|
|
140 .. option:: cxx_library_root=<path/to/lib/>
|
|
|
141
|
|
|
142 Specify the directory of the libc++ library to be tested. By default the
|
|
173
|
143 library folder of the build directory is used.
|
|
150
|
144
|
|
|
145
|
|
|
146 .. option:: cxx_runtime_root=<path/to/lib/>
|
|
|
147
|
|
|
148 Specify the directory of the libc++ library to use at runtime. This directory
|
|
|
149 is not added to the linkers search path. This can be used to compile tests
|
|
|
150 against one version of libc++ and run them using another. The default value
|
|
|
151 for this option is `cxx_library_root`.
|
|
|
152
|
|
|
153 .. option:: use_system_cxx_lib=<bool>
|
|
|
154
|
|
|
155 **Default**: False
|
|
|
156
|
|
|
157 Enable or disable testing against the installed version of libc++ library.
|
|
173
|
158 This impacts whether the ``with_system_cxx_lib`` Lit feature is defined or
|
|
|
159 not. The ``cxx_library_root`` and ``cxx_runtime_root`` parameters should
|
|
|
160 still be used to specify the path of the library to link to and run against,
|
|
|
161 respectively.
|
|
150
|
162
|
|
|
163 .. option:: use_lit_shell=<bool>
|
|
|
164
|
|
|
165 Enable or disable the use of LIT's internal shell in ShTests. If the
|
|
|
166 environment variable LIT_USE_INTERNAL_SHELL is present then that is used as
|
|
|
167 the default value. Otherwise the default value is True on Windows and False
|
|
|
168 on every other platform.
|
|
|
169
|
|
|
170 .. option:: compile_flags="<list-of-args>"
|
|
|
171
|
|
|
172 Specify additional compile flags as a space delimited string.
|
|
|
173 Note: This options should not be used to change the standard version used.
|
|
|
174
|
|
|
175 .. option:: link_flags="<list-of-args>"
|
|
|
176
|
|
|
177 Specify additional link flags as a space delimited string.
|
|
|
178
|
|
|
179 .. option:: debug_level=<level>
|
|
|
180
|
|
|
181 **Values**: 0, 1
|
|
|
182
|
|
|
183 Enable the use of debug mode. Level 0 enables assertions and level 1 enables
|
|
|
184 assertions and debugging of iterator misuse.
|
|
|
185
|
|
|
186 .. option:: use_sanitizer=<sanitizer name>
|
|
|
187
|
|
|
188 **Values**: Memory, MemoryWithOrigins, Address, Undefined
|
|
|
189
|
|
|
190 Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when
|
|
|
191 building libc++ then that sanitizer will be used by default.
|
|
|
192
|
|
|
193 .. option:: llvm_unwinder
|
|
|
194
|
|
|
195 Enable the use of LLVM unwinder instead of libgcc.
|
|
|
196
|
|
|
197 .. option:: builtins_library
|
|
|
198
|
|
|
199 Path to the builtins library to use instead of libgcc.
|
|
|
200
|
|
|
201
|
|
|
202 Environment Variables
|
|
|
203 ---------------------
|
|
|
204
|
|
|
205 .. envvar:: LIBCXX_SITE_CONFIG=<path/to/lit.site.cfg>
|
|
|
206
|
|
|
207 Specify the site configuration to use when running the tests.
|
|
|
208 Also see `libcxx_site_config`.
|
|
|
209
|
|
173
|
210 Writing Tests
|
|
|
211 -------------
|
|
|
212
|
|
|
213 When writing tests for the libc++ test suite, you should follow a few guidelines.
|
|
|
214 This will ensure that your tests can run on a wide variety of hardware and under
|
|
|
215 a wide variety of configurations. We have several unusual configurations such as
|
|
|
216 building the tests on one host but running them on a different host, which add a
|
|
|
217 few requirements to the test suite. Here's some stuff you should know:
|
|
150
|
218
|
|
173
|
219 - All tests are run in a temporary directory that is unique to that test and
|
|
|
220 cleaned up after the test is done.
|
|
|
221 - When a test needs data files as inputs, these data files can be saved in the
|
|
|
222 repository (when reasonable) and referrenced by the test as
|
|
|
223 ``// FILE_DEPENDENCIES: <path-to-dependencies>``. Copies of these files or
|
|
|
224 directories will be made available to the test in the temporary directory
|
|
|
225 where it is run.
|
|
|
226 - You should never hardcode a path from the build-host in a test, because that
|
|
|
227 path will not necessarily be available on the host where the tests are run.
|
|
|
228 - You should try to reduce the runtime dependencies of each test to the minimum.
|
|
|
229 For example, requiring Python to run a test is bad, since Python is not
|
|
|
230 necessarily available on all devices we may want to run the tests on (even
|
|
|
231 though supporting Python is probably trivial for the build-host).
|
|
150
|
232
|
|
|
233 Benchmarks
|
|
|
234 ==========
|
|
|
235
|
|
|
236 Libc++ contains benchmark tests separately from the test of the test suite.
|
|
|
237 The benchmarks are written using the `Google Benchmark`_ library, a copy of which
|
|
|
238 is stored in the libc++ repository.
|
|
|
239
|
|
|
240 For more information about using the Google Benchmark library see the
|
|
|
241 `official documentation <https://github.com/google/benchmark>`_.
|
|
|
242
|
|
|
243 .. _`Google Benchmark`: https://github.com/google/benchmark
|
|
|
244
|
|
|
245 Building Benchmarks
|
|
|
246 -------------------
|
|
|
247
|
|
|
248 The benchmark tests are not built by default. The benchmarks can be built using
|
|
|
249 the ``cxx-benchmarks`` target.
|
|
|
250
|
|
|
251 An example build would look like:
|
|
|
252
|
|
|
253 .. code-block:: bash
|
|
|
254
|
|
|
255 $ cd build
|
|
|
256 $ cmake [options] <path to libcxx sources>
|
|
|
257 $ make cxx-benchmarks
|
|
|
258
|
|
|
259 This will build all of the benchmarks under ``<libcxx-src>/benchmarks`` to be
|
|
|
260 built against the just-built libc++. The compiled tests are output into
|
|
|
261 ``build/benchmarks``.
|
|
|
262
|
|
|
263 The benchmarks can also be built against the platforms native standard library
|
|
|
264 using the ``-DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON`` CMake option. This
|
|
|
265 is useful for comparing the performance of libc++ to other standard libraries.
|
|
|
266 The compiled benchmarks are named ``<test>.libcxx.out`` if they test libc++ and
|
|
|
267 ``<test>.native.out`` otherwise.
|
|
|
268
|
|
|
269 Also See:
|
|
|
270
|
|
|
271 * :ref:`Building Libc++ <build instructions>`
|
|
|
272 * :ref:`CMake Options`
|
|
|
273
|
|
|
274 Running Benchmarks
|
|
|
275 ------------------
|
|
|
276
|
|
|
277 The benchmarks must be run manually by the user. Currently there is no way
|
|
|
278 to run them as part of the build.
|
|
|
279
|
|
|
280 For example:
|
|
|
281
|
|
|
282 .. code-block:: bash
|
|
|
283
|
|
|
284 $ cd build/benchmarks
|
|
|
285 $ make cxx-benchmarks
|
|
|
286 $ ./algorithms.libcxx.out # Runs all the benchmarks
|
|
|
287 $ ./algorithms.libcxx.out --benchmark_filter=BM_Sort.* # Only runs the sort benchmarks
|
|
|
288
|
|
|
289 For more information about running benchmarks see `Google Benchmark`_.
|
