diff libcxx/docs/TestingLibcxx.rst @ 173:0572611fdcc8 llvm10 llvm12

reorgnization done
author Shinji KONO <kono@ie.u-ryukyu.ac.jp>
date Mon, 25 May 2020 11:55:54 +0900
parents 1d019706d866
children 2e18cbf3894f
line wrap: on
line diff
--- a/libcxx/docs/TestingLibcxx.rst	Mon May 25 11:50:15 2020 +0900
+++ b/libcxx/docs/TestingLibcxx.rst	Mon May 25 11:55:54 2020 +0900
@@ -8,9 +8,9 @@
 Getting Started
 ===============
 
-libc++ uses LIT to configure and run its tests. 
+libc++ uses LIT to configure and run its tests.
 
-The primary way to run the libc++ tests is by using `make check-libcxx`. 
+The primary way to run the libc++ tests is by using ``make check-cxx``.
 
 However since libc++ can be used in any number of possible
 configurations it is important to customize the way LIT builds and runs
@@ -19,40 +19,23 @@
 
 Please see the `Lit Command Guide`_ for more information about LIT.
 
-.. _LIT Command Guide: http://llvm.org/docs/CommandGuide/lit.html
-
-Setting up the Environment
---------------------------
+.. _LIT Command Guide: https://llvm.org/docs/CommandGuide/lit.html
 
-After building libc++ you must setup your environment to test libc++ using
-LIT.
-
-#. Create a shortcut to the actual lit executable so that you can invoke it
-   easily from the command line.
-
-   .. code-block:: bash
+Usage
+-----
 
-     $ alias lit='python path/to/llvm/utils/lit/lit.py'
-
-#. Tell LIT where to find your build configuration.
-
-   .. code-block:: bash
-
-     $ export LIBCXX_SITE_CONFIG=path/to/build-libcxx/test/lit.site.cfg
-
-Example Usage
--------------
-
-Once you have your environment set up and you have built libc++ you can run
-parts of the libc++ test suite by simply running `lit` on a specified test or
-directory. For example:
+After building libc++, you can run parts of the libc++ test suite by simply
+running ``llvm-lit`` on a specified test or directory. If you're unsure
+whether the required libraries have been built, you can use the
+`check-cxx-deps` target. For example:
 
 .. code-block:: bash
 
-  $ cd path/to/src/libcxx
-  $ lit -sv test/std/re # Run all of the std::regex tests
-  $ lit -sv test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test
-  $ lit -sv test/std/atomics test/std/threads # Test std::thread and std::atomic
+  $ cd <monorepo-root>
+  $ make -C <build> check-cxx-deps # If you want to make sure the targets get rebuilt
+  $ <build>/bin/llvm-lit -sv libcxx/test/std/re # Run all of the std::regex tests
+  $ <build>/bin/llvm-lit -sv libcxx/test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test
+  $ <build>/bin/llvm-lit -sv libcxx/test/std/atomics libcxx/test/std/threads # Test std::thread and std::atomic
 
 Sometimes you'll want to change the way LIT is running the tests. Custom options
 can be specified using the `--param=<name>=<val>` flag. The most common option
@@ -62,30 +45,54 @@
 
 .. code-block:: bash
 
-  $ lit -sv test/std/containers # Run the tests with the newest -std
-  $ lit -sv --param=std=c++03 test/std/containers # Run the tests in C++03
+  $ <build>/bin/llvm-lit -sv libcxx/test/std/containers # Run the tests with the newest -std
+  $ <build>/bin/llvm-lit -sv libcxx/test/std/containers --param=std=c++03 # Run the tests in C++03
 
 Occasionally you'll want to add extra compile or link flags when testing.
 You can do this as follows:
 
 .. code-block:: bash
 
-  $ lit -sv --param=compile_flags='-Wcustom-warning'
-  $ lit -sv --param=link_flags='-L/custom/library/path'
+  $ <build>/bin/llvm-lit -sv libcxx/test --param=compile_flags='-Wcustom-warning'
+  $ <build>/bin/llvm-lit -sv libcxx/test --param=link_flags='-L/custom/library/path'
 
 Some other common examples include:
 
 .. code-block:: bash
 
   # Specify a custom compiler.
-  $ lit -sv --param=cxx_under_test=/opt/bin/g++ test/std
+  $ <build>/bin/llvm-lit -sv libcxx/test/std --param=cxx_under_test=/opt/bin/g++
 
   # Enable warnings in the test suite
-  $ lit -sv --param=enable_warnings=true test/std
+  $ <build>/bin/llvm-lit -sv libcxx/test --param=enable_warnings=true
 
   # Use UBSAN when running the tests.
-  $ lit -sv --param=use_sanitizer=Undefined
+  $ <build>/bin/llvm-lit -sv libcxx/test --param=use_sanitizer=Undefined
+
+Using a custom site configuration
+---------------------------------
 
+By default, the libc++ test suite will use a site configuration that matches
+the current CMake configuration. It does so by generating a ``lit.site.cfg``
+file in the build directory from the ``libcxx/test/lit.site.cfg.in`` template,
+and pointing ``llvm-lit`` (which is a wrapper around ``llvm/utils/lit/lit.py``)
+to that file. So when you're running ``<build>/bin/llvm-lit``, the generated
+``lit.site.cfg`` file is always loaded first, followed by the actual config in
+``libcxx/test/lit.cfg``. However, it is sometimes desirable to use a custom
+site configuration. To do that, you can use ``--param=libcxx_site_config`` or
+the ``LIBCXX_SITE_CONFIG`` environment variable to point to the right site
+configuration file. However, you must stop using ``llvm-lit``, or else the
+generated ``lit.site.cfg`` will still be preferred:
+
+   .. code-block:: bash
+
+     $ LIBCXX_SITE_CONFIG=path/to/your/site/configuration llvm/utils/lit/lit.py -sv ...
+
+     $ llvm/utils/lit/lit.py -sv ... --param=libcxx_site_config=path/to/your/site/configuration
+
+In both of these cases, your custom site configuration should set up the
+``config`` object in a way that is compatible with what libc++'s ``config.py``
+module expects.
 
 LIT Options
 ===========
@@ -95,9 +102,10 @@
 Command Line Options
 --------------------
 
-To use these options you pass them on the LIT command line as --param NAME or
---param NAME=VALUE. Some options have default values specified during CMake's
-configuration. Passing the option on the command line will override the default.
+To use these options you pass them on the LIT command line as ``--param NAME``
+or ``--param NAME=VALUE``. Some options have default values specified during
+CMake's configuration. Passing the option on the command line will override the
+default.
 
 .. program:: lit
 
@@ -132,8 +140,7 @@
 .. option:: cxx_library_root=<path/to/lib/>
 
   Specify the directory of the libc++ library to be tested. By default the
-  library folder of the build directory is used. This option cannot be used
-  when use_system_cxx_lib is provided.
+  library folder of the build directory is used.
 
 
 .. option:: cxx_runtime_root=<path/to/lib/>
@@ -148,7 +155,10 @@
   **Default**: False
 
   Enable or disable testing against the installed version of libc++ library.
-  Note: This does not use the installed headers.
+  This impacts whether the ``with_system_cxx_lib`` Lit feature is defined or
+  not. The ``cxx_library_root`` and ``cxx_runtime_root`` parameters should
+  still be used to specify the path of the library to link to and run against,
+  respectively.
 
 .. option:: use_lit_shell=<bool>
 
@@ -180,12 +190,6 @@
   Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when
   building libc++ then that sanitizer will be used by default.
 
-.. option:: color_diagnostics
-
-  Enable the use of colorized compile diagnostics. If the color_diagnostics
-  option is specified or the environment variable LIBCXX_COLOR_DIAGNOSTICS is
-  present then color diagnostics will be enabled.
-
 .. option:: llvm_unwinder
 
   Enable the use of LLVM unwinder instead of libgcc.
@@ -203,11 +207,28 @@
   Specify the site configuration to use when running the tests.
   Also see `libcxx_site_config`.
 
-.. envvar:: LIBCXX_COLOR_DIAGNOSTICS
+Writing Tests
+-------------
+
+When writing tests for the libc++ test suite, you should follow a few guidelines.
+This will ensure that your tests can run on a wide variety of hardware and under
+a wide variety of configurations. We have several unusual configurations such as
+building the tests on one host but running them on a different host, which add a
+few requirements to the test suite. Here's some stuff you should know:
 
-  If ``LIBCXX_COLOR_DIAGNOSTICS`` is defined then the test suite will attempt
-  to use color diagnostic outputs from the compiler.
-  Also see `color_diagnostics`.
+- All tests are run in a temporary directory that is unique to that test and
+  cleaned up after the test is done.
+- When a test needs data files as inputs, these data files can be saved in the
+  repository (when reasonable) and referrenced by the test as
+  ``// FILE_DEPENDENCIES: <path-to-dependencies>``. Copies of these files or
+  directories will be made available to the test in the temporary directory
+  where it is run.
+- You should never hardcode a path from the build-host in a test, because that
+  path will not necessarily be available on the host where the tests are run.
+- You should try to reduce the runtime dependencies of each test to the minimum.
+  For example, requiring Python to run a test is bad, since Python is not
+  necessarily available on all devices we may want to run the tests on (even
+  though supporting Python is probably trivial for the build-host).
 
 Benchmarks
 ==========