Mercurial > hg > CbC > CbC_llvm

comparison docs/TestingGuide.rst @ 147:c2174574ed3a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
LLVM 10
author Shinji KONO <kono@ie.u-ryukyu.ac.jp>
date Wed, 14 Aug 2019 16:55:33 +0900
parents 803732b1fca8
children
comparison
equal deleted inserted replaced
134:3a76565eade5 147:c2174574ed3a
6 :local: 6 :local:
7 7
8 .. toctree:: 8 .. toctree::
9 :hidden: 9 :hidden:
10 10
11 TestSuiteGuide
11 TestSuiteMakefileGuide 12 TestSuiteMakefileGuide
12 13
13 Overview 14 Overview
14 ======== 15 ========
15 16
23 24
24 In order to use the LLVM testing infrastructure, you will need all of the 25 In order to use the LLVM testing infrastructure, you will need all of the
25 software required to build LLVM, as well as `Python <http://python.org>`_ 2.7 or 26 software required to build LLVM, as well as `Python <http://python.org>`_ 2.7 or
26 later. 27 later.
27 28
28 If you intend to run the :ref:`test-suite <test-suite-overview>`, you will also 29 LLVM Testing Infrastructure Organization
29 need a development version of zlib (zlib1g-dev is known to work on several Linux
30 distributions).
31
32 LLVM testing infrastructure organization
33 ======================================== 30 ========================================
34 31
35 The LLVM testing infrastructure contains two major categories of tests: 32 The LLVM testing infrastructure contains three major categories of tests:
36 regression tests and whole programs. The regression tests are contained 33 unit tests, regression tests and whole programs. The unit tests and regression
37 inside the LLVM repository itself under ``llvm/test`` and are expected 34 tests are contained inside the LLVM repository itself under ``llvm/unittests``
38 to always pass -- they should be run before every commit. 35 and ``llvm/test`` respectively and are expected to always pass -- they should be
36 run before every commit.
39 37
40 The whole programs tests are referred to as the "LLVM test suite" (or 38 The whole programs tests are referred to as the "LLVM test suite" (or
41 "test-suite") and are in the ``test-suite`` module in subversion. For 39 "test-suite") and are in the ``test-suite`` module in subversion. For
42 historical reasons, these tests are also referred to as the "nightly 40 historical reasons, these tests are also referred to as the "nightly
43 tests" in places, which is less ambiguous than "test-suite" and remains 41 tests" in places, which is less ambiguous than "test-suite" and remains
44 in use although we run them much more often than nightly. 42 in use although we run them much more often than nightly.
45 43
44 Unit tests
45 ----------
46
47 Unit tests are written using `Google Test <https://github.com/google/googletest/blob/master/googletest/docs/primer.md>`_
48 and `Google Mock <https://github.com/google/googletest/blob/master/googlemock/docs/ForDummies.md>`_
49 and are located in the ``llvm/unittests`` directory.
50
46 Regression tests 51 Regression tests
47 ---------------- 52 ----------------
48 53
49 The regression tests are small pieces of code that test a specific 54 The regression tests are small pieces of code that test a specific
50 feature of LLVM or trigger a specific bug in LLVM. The language they are 55 feature of LLVM or trigger a specific bug in LLVM. The language they are
75 efficiency of the programs generated as well as the speed with which 80 efficiency of the programs generated as well as the speed with which
76 LLVM compiles, optimizes, and generates code. 81 LLVM compiles, optimizes, and generates code.
77 82
78 The test-suite is located in the ``test-suite`` Subversion module. 83 The test-suite is located in the ``test-suite`` Subversion module.
79 84
85 See the :doc:`TestSuiteGuide` for details.
86
80 Debugging Information tests 87 Debugging Information tests
81 --------------------------- 88 ---------------------------
82 89
83 The test suite contains tests to check quality of debugging information. 90 The test suite contains tests to check quality of debugging information.
84 The test are written in C based languages or in LLVM assembly language. 91 The test are written in C based languages or in LLVM assembly language.
89 ``debuginfo-tests`` Subversion module. 96 ``debuginfo-tests`` Subversion module.
90 97
91 Quick start 98 Quick start
92 =========== 99 ===========
93 100
94 The tests are located in two separate Subversion modules. The 101 The tests are located in two separate Subversion modules. The unit and
95 regressions tests are in the main "llvm" module under the directory 102 regression tests are in the main "llvm" module under the directories
96 ``llvm/test`` (so you get these tests for free with the main LLVM tree). 103 ``llvm/unittests`` and ``llvm/test`` (so you get these tests for free with the
97 Use ``make check-all`` to run the regression tests after building LLVM. 104 main LLVM tree). Use ``make check-all`` to run the unit and regression tests
98 105 after building LLVM.
99 The more comprehensive test suite that includes whole programs in C and C++ 106
100 is in the ``test-suite`` module. See :ref:`test-suite Quickstart 107 The ``test-suite`` module contains more comprehensive tests including whole C
101 <test-suite-quickstart>` for more information on running these tests. 108 and C++ programs. See the :doc:`TestSuiteGuide` for details.
102 109
103 Regression tests 110 Unit and Regression tests
104 ---------------- 111 -------------------------
112
113 To run all of the LLVM unit tests use the check-llvm-unit target:
114
115 .. code-block:: bash
116
117 % make check-llvm-unit
105 118
106 To run all of the LLVM regression tests use the check-llvm target: 119 To run all of the LLVM regression tests use the check-llvm target:
107 120
108 .. code-block:: bash 121 .. code-block:: bash
109 122
110 % make check-llvm 123 % make check-llvm
124
125 In order to get reasonable testing performance, build LLVM and subprojects
126 in release mode, i.e.
127
128 .. code-block:: bash
129
130 % cmake -DCMAKE_BUILD_TYPE="Release" -DLLVM_ENABLE_ASSERTIONS=On
111 131
112 If you have `Clang <http://clang.llvm.org/>`_ checked out and built, you 132 If you have `Clang <http://clang.llvm.org/>`_ checked out and built, you
113 can run the LLVM and Clang tests simultaneously using: 133 can run the LLVM and Clang tests simultaneously using:
114 134
115 .. code-block:: bash 135 .. code-block:: bash
143 or the :doc:`lit man page <CommandGuide/lit>`. 163 or the :doc:`lit man page <CommandGuide/lit>`.
144 164
145 Debugging Information tests 165 Debugging Information tests
146 --------------------------- 166 ---------------------------
147 167
148 To run debugging information tests simply checkout the tests inside 168 To run debugging information tests simply add the ``debuginfo-tests``
149 clang/test directory. 169 project to your ``LLVM_ENABLE_PROJECTS`` define on the cmake
150 170 command-line.
151 .. code-block:: bash
152
153 % cd clang/test
154 % svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests
155
156 These tests are already set up to run as part of clang regression tests.
157 171
158 Regression test structure 172 Regression test structure
159 ========================= 173 =========================
160 174
161 The LLVM regression tests are driven by :program:`lit` and are located in the 175 The LLVM regression tests are driven by :program:`lit` and are located in the
458 some redirected output. 472 some redirected output.
459 473
460 Example: ``/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp`` 474 Example: ``/home/user/llvm.build/test/MC/ELF/Output/foo_test.s.tmp``
461 475
462 ``%T`` 476 ``%T``
463 Directory of ``%t``. 477 Directory of ``%t``. Deprecated. Shouldn't be used, because it can be easily
478 misused and cause race conditions between tests.
479
480 Use ``rm -rf %t && mkdir %t`` instead if a temporary directory is necessary.
464 481
465 Example: ``/home/user/llvm.build/test/MC/ELF/Output`` 482 Example: ``/home/user/llvm.build/test/MC/ELF/Output``
466 483
467 ``%{pathsep}`` 484 ``%{pathsep}``
468 485
492 509
493 ``%shlibext`` 510 ``%shlibext``
494 The suffix for the host platforms shared library files. This includes the 511 The suffix for the host platforms shared library files. This includes the
495 period as the first character. 512 period as the first character.
496 513
497 Example: ``.so`` (Linux), ``.dylib`` (OS X), ``.dll`` (Windows) 514 Example: ``.so`` (Linux), ``.dylib`` (macOS), ``.dll`` (Windows)
498 515
499 ``%exeext`` 516 ``%exeext``
500 The suffix for the host platforms executable files. This includes the 517 The suffix for the host platforms executable files. This includes the
501 period as the first character. 518 period as the first character.
502 519
580 (a) it prevents special interpretation of lines that are part of the test 597 (a) it prevents special interpretation of lines that are part of the test
581 program, not the instructions to the test case, and 598 program, not the instructions to the test case, and
582 599
583 (b) it speeds things up for really big test cases by avoiding 600 (b) it speeds things up for really big test cases by avoiding
584 interpretation of the remainder of the file. 601 interpretation of the remainder of the file.
585
586 .. _test-suite-overview:
587
588 ``test-suite`` Overview
589 =======================
590
591 The ``test-suite`` module contains a number of programs that can be
592 compiled and executed. The ``test-suite`` includes reference outputs for
593 all of the programs, so that the output of the executed program can be
594 checked for correctness.
595
596 ``test-suite`` tests are divided into three types of tests: MultiSource,
597 SingleSource, and External.
598
599 - ``test-suite/SingleSource``
600
601 The SingleSource directory contains test programs that are only a
602 single source file in size. These are usually small benchmark
603 programs or small programs that calculate a particular value. Several
604 such programs are grouped together in each directory.
605
606 - ``test-suite/MultiSource``
607
608 The MultiSource directory contains subdirectories which contain
609 entire programs with multiple source files. Large benchmarks and
610 whole applications go here.
611
612 - ``test-suite/External``
613
614 The External directory contains Makefiles for building code that is
615 external to (i.e., not distributed with) LLVM. The most prominent
616 members of this directory are the SPEC 95 and SPEC 2000 benchmark
617 suites. The ``External`` directory does not contain these actual
618 tests, but only the Makefiles that know how to properly compile these
619 programs from somewhere else. When using ``LNT``, use the
620 ``--test-externals`` option to include these tests in the results.
621
622 .. _test-suite-quickstart:
623
624 ``test-suite`` Quickstart
625 -------------------------
626
627 The modern way of running the ``test-suite`` is focused on testing and
628 benchmarking complete compilers using the
629 `LNT <http://llvm.org/docs/lnt>`_ testing infrastructure.
630
631 For more information on using LNT to execute the ``test-suite``, please
632 see the `LNT Quickstart <http://llvm.org/docs/lnt/quickstart.html>`_
633 documentation.
634
635 ``test-suite`` Makefiles
636 ------------------------
637
638 Historically, the ``test-suite`` was executed using a complicated setup
639 of Makefiles. The LNT based approach above is recommended for most
640 users, but there are some testing scenarios which are not supported by
641 the LNT approach. In addition, LNT currently uses the Makefile setup
642 under the covers and so developers who are interested in how LNT works
643 under the hood may want to understand the Makefile based setup.
644
645 For more information on the ``test-suite`` Makefile setup, please see
646 the :doc:`Test Suite Makefile Guide <TestSuiteMakefileGuide>`.