Mercurial > hg > CbC > CbC_llvm

comparison libcxx/docs/TestingLibcxx.rst @ 173:0572611fdcc8 llvm10 llvm12

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
reorgnization done
author Shinji KONO <kono@ie.u-ryukyu.ac.jp>
date Mon, 25 May 2020 11:55:54 +0900
parents 1d019706d866
children 2e18cbf3894f
comparison
equal deleted inserted replaced
172:9fbae9c8bf63 173:0572611fdcc8
6 :local: 6 :local:
7 7
8 Getting Started 8 Getting Started
9 =============== 9 ===============
10 10
11 libc++ uses LIT to configure and run its tests. 11 libc++ uses LIT to configure and run its tests.
12 12
13 The primary way to run the libc++ tests is by using `make check-libcxx`. 13 The primary way to run the libc++ tests is by using ``make check-cxx``.
14 14
15 However since libc++ can be used in any number of possible 15 However since libc++ can be used in any number of possible
16 configurations it is important to customize the way LIT builds and runs 16 configurations it is important to customize the way LIT builds and runs
17 the tests. This guide provides information on how to use LIT directly to 17 the tests. This guide provides information on how to use LIT directly to
18 test libc++. 18 test libc++.
19 19
20 Please see the `Lit Command Guide`_ for more information about LIT. 20 Please see the `Lit Command Guide`_ for more information about LIT.
21 21
22 .. _LIT Command Guide: http://llvm.org/docs/CommandGuide/lit.html 22 .. _LIT Command Guide: https://llvm.org/docs/CommandGuide/lit.html
23 23
24 Setting up the Environment 24 Usage
25 -------------------------- 25 -----
26 26
27 After building libc++ you must setup your environment to test libc++ using 27 After building libc++, you can run parts of the libc++ test suite by simply
28 LIT. 28 running ``llvm-lit`` on a specified test or directory. If you're unsure
29 29 whether the required libraries have been built, you can use the
30 #. Create a shortcut to the actual lit executable so that you can invoke it 30 `check-cxx-deps` target. For example:
31 easily from the command line. 31
32 32 .. code-block:: bash
33 .. code-block:: bash 33
34 34 $ cd <monorepo-root>
35 $ alias lit='python path/to/llvm/utils/lit/lit.py' 35 $ make -C <build> check-cxx-deps # If you want to make sure the targets get rebuilt
36 36 $ <build>/bin/llvm-lit -sv libcxx/test/std/re # Run all of the std::regex tests
37 #. Tell LIT where to find your build configuration. 37 $ <build>/bin/llvm-lit -sv libcxx/test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test
38 38 $ <build>/bin/llvm-lit -sv libcxx/test/std/atomics libcxx/test/std/threads # Test std::thread and std::atomic
39 .. code-block:: bash
40
41 $ export LIBCXX_SITE_CONFIG=path/to/build-libcxx/test/lit.site.cfg
42
43 Example Usage
44 -------------
45
46 Once you have your environment set up and you have built libc++ you can run
47 parts of the libc++ test suite by simply running `lit` on a specified test or
48 directory. For example:
49
50 .. code-block:: bash
51
52 $ cd path/to/src/libcxx
53 $ lit -sv test/std/re # Run all of the std::regex tests
54 $ lit -sv test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test
55 $ lit -sv test/std/atomics test/std/threads # Test std::thread and std::atomic
56 39
57 Sometimes you'll want to change the way LIT is running the tests. Custom options 40 Sometimes you'll want to change the way LIT is running the tests. Custom options
58 can be specified using the `--param=<name>=<val>` flag. The most common option 41 can be specified using the `--param=<name>=<val>` flag. The most common option
59 you'll want to change is the standard dialect (ie -std=c++XX). By default the 42 you'll want to change is the standard dialect (ie -std=c++XX). By default the
60 test suite will select the newest C++ dialect supported by the compiler and use 43 test suite will select the newest C++ dialect supported by the compiler and use
61 that. However if you want to manually specify the option like so: 44 that. However if you want to manually specify the option like so:
62 45
63 .. code-block:: bash 46 .. code-block:: bash
64 47
65 $ lit -sv test/std/containers # Run the tests with the newest -std 48 $ <build>/bin/llvm-lit -sv libcxx/test/std/containers # Run the tests with the newest -std
66 $ lit -sv --param=std=c++03 test/std/containers # Run the tests in C++03 49 $ <build>/bin/llvm-lit -sv libcxx/test/std/containers --param=std=c++03 # Run the tests in C++03
67 50
68 Occasionally you'll want to add extra compile or link flags when testing. 51 Occasionally you'll want to add extra compile or link flags when testing.
69 You can do this as follows: 52 You can do this as follows:
70 53
71 .. code-block:: bash 54 .. code-block:: bash
72 55
73 $ lit -sv --param=compile_flags='-Wcustom-warning' 56 $ <build>/bin/llvm-lit -sv libcxx/test --param=compile_flags='-Wcustom-warning'
74 $ lit -sv --param=link_flags='-L/custom/library/path' 57 $ <build>/bin/llvm-lit -sv libcxx/test --param=link_flags='-L/custom/library/path'
75 58
76 Some other common examples include: 59 Some other common examples include:
77 60
78 .. code-block:: bash 61 .. code-block:: bash
79 62
80 # Specify a custom compiler. 63 # Specify a custom compiler.
81 $ lit -sv --param=cxx_under_test=/opt/bin/g++ test/std 64 $ <build>/bin/llvm-lit -sv libcxx/test/std --param=cxx_under_test=/opt/bin/g++
82 65
83 # Enable warnings in the test suite 66 # Enable warnings in the test suite
84 $ lit -sv --param=enable_warnings=true test/std 67 $ <build>/bin/llvm-lit -sv libcxx/test --param=enable_warnings=true
85 68
86 # Use UBSAN when running the tests. 69 # Use UBSAN when running the tests.
87 $ lit -sv --param=use_sanitizer=Undefined 70 $ <build>/bin/llvm-lit -sv libcxx/test --param=use_sanitizer=Undefined
88 71
72 Using a custom site configuration
73 ---------------------------------
74
75 By default, the libc++ test suite will use a site configuration that matches
76 the current CMake configuration. It does so by generating a ``lit.site.cfg``
77 file in the build directory from the ``libcxx/test/lit.site.cfg.in`` template,
78 and pointing ``llvm-lit`` (which is a wrapper around ``llvm/utils/lit/lit.py``)
79 to that file. So when you're running ``<build>/bin/llvm-lit``, the generated
80 ``lit.site.cfg`` file is always loaded first, followed by the actual config in
81 ``libcxx/test/lit.cfg``. However, it is sometimes desirable to use a custom
82 site configuration. To do that, you can use ``--param=libcxx_site_config`` or
83 the ``LIBCXX_SITE_CONFIG`` environment variable to point to the right site
84 configuration file. However, you must stop using ``llvm-lit``, or else the
85 generated ``lit.site.cfg`` will still be preferred:
86
87 .. code-block:: bash
88
89 $ LIBCXX_SITE_CONFIG=path/to/your/site/configuration llvm/utils/lit/lit.py -sv ...
90
91 $ llvm/utils/lit/lit.py -sv ... --param=libcxx_site_config=path/to/your/site/configuration
92
93 In both of these cases, your custom site configuration should set up the
94 ``config`` object in a way that is compatible with what libc++'s ``config.py``
95 module expects.
89 96
90 LIT Options 97 LIT Options
91 =========== 98 ===========
92 99
93 :program:`lit` [*options*...] [*filenames*...] 100 :program:`lit` [*options*...] [*filenames*...]
94 101
95 Command Line Options 102 Command Line Options
96 -------------------- 103 --------------------
97 104
98 To use these options you pass them on the LIT command line as --param NAME or 105 To use these options you pass them on the LIT command line as ``--param NAME``
99 --param NAME=VALUE. Some options have default values specified during CMake's 106 or ``--param NAME=VALUE``. Some options have default values specified during
100 configuration. Passing the option on the command line will override the default. 107 CMake's configuration. Passing the option on the command line will override the
108 default.
101 109
102 .. program:: lit 110 .. program:: lit
103 111
104 .. option:: cxx_under_test=<path/to/compiler> 112 .. option:: cxx_under_test=<path/to/compiler>
105 113
130 headers in the source tree are used. 138 headers in the source tree are used.
131 139
132 .. option:: cxx_library_root=<path/to/lib/> 140 .. option:: cxx_library_root=<path/to/lib/>
133 141
134 Specify the directory of the libc++ library to be tested. By default the 142 Specify the directory of the libc++ library to be tested. By default the
135 library folder of the build directory is used. This option cannot be used 143 library folder of the build directory is used.
136 when use_system_cxx_lib is provided.
137 144
138 145
139 .. option:: cxx_runtime_root=<path/to/lib/> 146 .. option:: cxx_runtime_root=<path/to/lib/>
140 147
141 Specify the directory of the libc++ library to use at runtime. This directory 148 Specify the directory of the libc++ library to use at runtime. This directory
146 .. option:: use_system_cxx_lib=<bool> 153 .. option:: use_system_cxx_lib=<bool>
147 154
148 **Default**: False 155 **Default**: False
149 156
150 Enable or disable testing against the installed version of libc++ library. 157 Enable or disable testing against the installed version of libc++ library.
151 Note: This does not use the installed headers. 158 This impacts whether the ``with_system_cxx_lib`` Lit feature is defined or
159 not. The ``cxx_library_root`` and ``cxx_runtime_root`` parameters should
160 still be used to specify the path of the library to link to and run against,
161 respectively.
152 162
153 .. option:: use_lit_shell=<bool> 163 .. option:: use_lit_shell=<bool>
154 164
155 Enable or disable the use of LIT's internal shell in ShTests. If the 165 Enable or disable the use of LIT's internal shell in ShTests. If the
156 environment variable LIT_USE_INTERNAL_SHELL is present then that is used as 166 environment variable LIT_USE_INTERNAL_SHELL is present then that is used as
178 **Values**: Memory, MemoryWithOrigins, Address, Undefined 188 **Values**: Memory, MemoryWithOrigins, Address, Undefined
179 189
180 Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when 190 Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when
181 building libc++ then that sanitizer will be used by default. 191 building libc++ then that sanitizer will be used by default.
182 192
183 .. option:: color_diagnostics
184
185 Enable the use of colorized compile diagnostics. If the color_diagnostics
186 option is specified or the environment variable LIBCXX_COLOR_DIAGNOSTICS is
187 present then color diagnostics will be enabled.
188
189 .. option:: llvm_unwinder 193 .. option:: llvm_unwinder
190 194
191 Enable the use of LLVM unwinder instead of libgcc. 195 Enable the use of LLVM unwinder instead of libgcc.
192 196
193 .. option:: builtins_library 197 .. option:: builtins_library
201 .. envvar:: LIBCXX_SITE_CONFIG=<path/to/lit.site.cfg> 205 .. envvar:: LIBCXX_SITE_CONFIG=<path/to/lit.site.cfg>
202 206
203 Specify the site configuration to use when running the tests. 207 Specify the site configuration to use when running the tests.
204 Also see `libcxx_site_config`. 208 Also see `libcxx_site_config`.
205 209
206 .. envvar:: LIBCXX_COLOR_DIAGNOSTICS 210 Writing Tests
207 211 -------------
208 If ``LIBCXX_COLOR_DIAGNOSTICS`` is defined then the test suite will attempt 212
209 to use color diagnostic outputs from the compiler. 213 When writing tests for the libc++ test suite, you should follow a few guidelines.
210 Also see `color_diagnostics`. 214 This will ensure that your tests can run on a wide variety of hardware and under
215 a wide variety of configurations. We have several unusual configurations such as
216 building the tests on one host but running them on a different host, which add a
217 few requirements to the test suite. Here's some stuff you should know:
218
219 - All tests are run in a temporary directory that is unique to that test and
220 cleaned up after the test is done.
221 - When a test needs data files as inputs, these data files can be saved in the
222 repository (when reasonable) and referrenced by the test as
223 ``// FILE_DEPENDENCIES: <path-to-dependencies>``. Copies of these files or
224 directories will be made available to the test in the temporary directory
225 where it is run.
226 - You should never hardcode a path from the build-host in a test, because that
227 path will not necessarily be available on the host where the tests are run.
228 - You should try to reduce the runtime dependencies of each test to the minimum.
229 For example, requiring Python to run a test is bad, since Python is not
230 necessarily available on all devices we may want to run the tests on (even
231 though supporting Python is probably trivial for the build-host).
211 232
212 Benchmarks 233 Benchmarks
213 ========== 234 ==========
214 235
215 Libc++ contains benchmark tests separately from the test of the test suite. 236 Libc++ contains benchmark tests separately from the test of the test suite.