|
150
|
1 =============================
|
|
|
2 How To Validate a New Release
|
|
|
3 =============================
|
|
|
4
|
|
|
5 .. contents::
|
|
|
6 :local:
|
|
|
7 :depth: 1
|
|
|
8
|
|
|
9 Introduction
|
|
|
10 ============
|
|
|
11
|
|
|
12 This document contains information about testing the release candidates that
|
|
|
13 will ultimately be the next LLVM release. For more information on how to
|
|
|
14 manage the actual release, please refer to :doc:`HowToReleaseLLVM`.
|
|
|
15
|
|
|
16 Overview of the Release Process
|
|
|
17 -------------------------------
|
|
|
18
|
|
|
19 Once the release process starts, the Release Manager will ask for volunteers,
|
|
|
20 and it'll be the role of each volunteer to:
|
|
|
21
|
|
|
22 * Test and benchmark the previous release
|
|
|
23
|
|
|
24 * Test and benchmark each release candidate, comparing to the previous release
|
|
|
25 and candidates
|
|
|
26
|
|
|
27 * Identify, reduce and report every regression found during tests and benchmarks
|
|
|
28
|
|
|
29 * Make sure the critical bugs get fixed and merged to the next release candidate
|
|
|
30
|
|
|
31 Not all bugs or regressions are show-stoppers and it's a bit of a grey area what
|
|
|
32 should be fixed before the next candidate and what can wait until the next
|
|
|
33 release.
|
|
|
34
|
|
|
35 It'll depend on:
|
|
|
36
|
|
|
37 * The severity of the bug, how many people it affects and if it's a regression
|
|
|
38 or a known bug. Known bugs are "unsupported features" and some bugs can be
|
|
|
39 disabled if they have been implemented recently.
|
|
|
40
|
|
|
41 * The stage in the release. Less critical bugs should be considered to be
|
|
|
42 fixed between RC1 and RC2, but not so much at the end of it.
|
|
|
43
|
|
|
44 * If it's a correctness or a performance regression. Performance regression
|
|
|
45 tends to be taken more lightly than correctness.
|
|
|
46
|
|
|
47 .. _scripts:
|
|
|
48
|
|
|
49 Scripts
|
|
|
50 =======
|
|
|
51
|
|
|
52 The scripts are in the ``utils/release`` directory.
|
|
|
53
|
|
|
54 test-release.sh
|
|
|
55 ---------------
|
|
|
56
|
|
|
57 This script will check-out, configure and compile LLVM+Clang (+ most add-ons,
|
|
|
58 like ``compiler-rt``, ``libcxx``, ``libomp`` and ``clang-extra-tools``) in
|
|
|
59 three stages, and will test the final stage.
|
|
|
60 It'll have installed the final binaries on the Phase3/Releasei(+Asserts)
|
|
|
61 directory, and that's the one you should use for the test-suite and other
|
|
|
62 external tests.
|
|
|
63
|
|
|
64 To run the script on a specific release candidate run::
|
|
|
65
|
|
|
66 ./test-release.sh \
|
|
|
67 -release 3.3 \
|
|
|
68 -rc 1 \
|
|
|
69 -no-64bit \
|
|
|
70 -test-asserts \
|
|
|
71 -no-compare-files
|
|
|
72
|
|
|
73 Each system will require different options. For instance, x86_64 will
|
|
|
74 obviously not need ``-no-64bit`` while 32-bit systems will, or the script will
|
|
|
75 fail.
|
|
|
76
|
|
|
77 The important flags to get right are:
|
|
|
78
|
|
|
79 * On the pre-release, you should change ``-rc 1`` to ``-final``. On RC2,
|
|
|
80 change it to ``-rc 2`` and so on.
|
|
|
81
|
|
|
82 * On non-release testing, you can use ``-final`` in conjunction with
|
|
|
83 ``-no-checkout``, but you'll have to create the ``final`` directory by hand
|
|
|
84 and link the correct source dir to ``final/llvm.src``.
|
|
|
85
|
|
|
86 * For release candidates, you need ``-test-asserts``, or it won't create a
|
|
|
87 "Release+Asserts" directory, which is needed for release testing and
|
|
|
88 benchmarking. This will take twice as long.
|
|
|
89
|
|
|
90 * On the final candidate you just need Release builds, and that's the binary
|
|
|
91 directory you'll have to pack.
|
|
|
92
|
|
|
93 * On macOS, you must export ``MACOSX_DEPLOYMENT_TARGET=10.9`` before running
|
|
|
94 the script.
|
|
|
95
|
|
|
96 This script builds three phases of Clang+LLVM twice each (Release and
|
|
|
97 Release+Asserts), so use screen or nohup to avoid headaches, since it'll take
|
|
|
98 a long time.
|
|
|
99
|
|
|
100 Use the ``--help`` option to see all the options and chose it according to
|
|
|
101 your needs.
|
|
|
102
|
|
|
103
|
|
|
104 findRegressions-nightly.py
|
|
|
105 --------------------------
|
|
|
106
|
|
|
107 TODO
|
|
|
108
|
|
|
109 .. _test-suite:
|
|
|
110
|
|
|
111 Test Suite
|
|
|
112 ==========
|
|
|
113
|
|
|
114 .. contents::
|
|
|
115 :local:
|
|
|
116
|
|
|
117 Follow the `LNT Quick Start Guide
|
|
173
|
118 <https://llvm.org/docs/lnt/quickstart.html>`__ link on how to set-up the
|
|
150
|
119 test-suite
|
|
|
120
|
|
|
121 The binary location you'll have to use for testing is inside the
|
|
|
122 ``rcN/Phase3/Release+Asserts/llvmCore-REL-RC.install``.
|
|
|
123 Link that directory to an easier location and run the test-suite.
|
|
|
124
|
|
|
125 An example on the run command line, assuming you created a link from the correct
|
|
|
126 install directory to ``~/devel/llvm/install``::
|
|
|
127
|
|
|
128 ./sandbox/bin/python sandbox/bin/lnt runtest \
|
|
|
129 nt \
|
|
|
130 -j4 \
|
|
|
131 --sandbox sandbox \
|
|
|
132 --test-suite ~/devel/llvm/test/test-suite \
|
|
|
133 --cc ~/devel/llvm/install/bin/clang \
|
|
|
134 --cxx ~/devel/llvm/install/bin/clang++
|
|
|
135
|
|
|
136 It should have no new regressions, compared to the previous release or release
|
|
|
137 candidate. You don't need to fix all the bugs in the test-suite, since they're
|
|
|
138 not necessarily meant to pass on all architectures all the time. This is
|
|
|
139 due to the nature of the result checking, which relies on direct comparison,
|
|
|
140 and most of the time, the failures are related to bad output checking, rather
|
|
|
141 than bad code generation.
|
|
|
142
|
|
|
143 If the errors are in LLVM itself, please report every single regression found
|
|
|
144 as blocker, and all the other bugs as important, but not necessarily blocking
|
|
|
145 the release to proceed. They can be set as "known failures" and to be
|
|
|
146 fix on a future date.
|
|
|
147
|
|
|
148 .. _pre-release-process:
|
|
|
149
|
|
|
150 Pre-Release Process
|
|
|
151 ===================
|
|
|
152
|
|
|
153 .. contents::
|
|
|
154 :local:
|
|
|
155
|
|
|
156 When the release process is announced on the mailing list, you should prepare
|
|
|
157 for the testing, by applying the same testing you'll do on the release
|
|
|
158 candidates, on the previous release.
|
|
|
159
|
|
|
160 You should:
|
|
|
161
|
|
|
162 * Download the previous release sources from
|
|
173
|
163 https://llvm.org/releases/download.html.
|
|
150
|
164
|
|
|
165 * Run the test-release.sh script on ``final`` mode (change ``-rc 1`` to
|
|
|
166 ``-final``).
|
|
|
167
|
|
|
168 * Once all three stages are done, it'll test the final stage.
|
|
|
169
|
|
|
170 * Using the ``Phase3/Release+Asserts/llvmCore-MAJ.MIN-final.install`` base,
|
|
|
171 run the test-suite.
|
|
|
172
|
|
|
173 If the final phase's ``make check-all`` failed, it's a good idea to also test
|
|
|
174 the intermediate stages by going on the obj directory and running
|
|
|
175 ``make check-all`` to find if there's at least one stage that passes (helps
|
|
|
176 when reducing the error for bug report purposes).
|
|
|
177
|
|
|
178 .. _release-process:
|
|
|
179
|
|
|
180 Release Process
|
|
|
181 ===============
|
|
|
182
|
|
|
183 .. contents::
|
|
|
184 :local:
|
|
|
185
|
|
|
186 When the Release Manager sends you the release candidate, download all sources,
|
|
|
187 unzip on the same directory (there will be sym-links from the appropriate places
|
|
|
188 to them), and run the release test as above.
|
|
|
189
|
|
|
190 You should:
|
|
|
191
|
|
|
192 * Download the current candidate sources from where the release manager points
|
|
173
|
193 you (ex. https://llvm.org/pre-releases/3.3/rc1/).
|
|
150
|
194
|
|
|
195 * Repeat the steps above with ``-rc 1``, ``-rc 2`` etc modes and run the
|
|
|
196 test-suite the same way.
|
|
|
197
|
|
|
198 * Compare the results, report all errors on Bugzilla and publish the binary blob
|
|
|
199 where the release manager can grab it.
|
|
|
200
|
|
|
201 Once the release manages announces that the latest candidate is the good one,
|
|
|
202 you have to pack the ``Release`` (no Asserts) install directory on ``Phase3``
|
|
|
203 and that will be the official binary.
|
|
|
204
|
|
|
205 * Rename (or link) ``clang+llvm-REL-ARCH-ENV`` to the .install directory
|
|
|
206
|
|
|
207 * Tar that into the same name with ``.tar.gz`` extension from outside the
|
|
|
208 directory
|
|
|
209
|
|
|
210 * Make it available for the release manager to download
|
|
|
211
|
|
|
212 .. _bug-reporting:
|
|
|
213
|
|
|
214 Bug Reporting Process
|
|
|
215 =====================
|
|
|
216
|
|
|
217 .. contents::
|
|
|
218 :local:
|
|
|
219
|
|
|
220 If you found regressions or failures when comparing a release candidate with the
|
|
|
221 previous release, follow the rules below:
|
|
|
222
|
|
|
223 * Critical bugs on compilation should be fixed as soon as possible, possibly
|
|
|
224 before releasing the binary blobs.
|
|
|
225
|
|
|
226 * Check-all tests should be fixed before the next release candidate, but can
|
|
|
227 wait until the test-suite run is finished.
|
|
|
228
|
|
|
229 * Bugs in the test suite or unimportant check-all tests can be fixed in between
|
|
|
230 release candidates.
|
|
|
231
|
|
|
232 * New features or recent big changes, when close to the release, should have
|
|
|
233 done in a way that it's easy to disable. If they misbehave, prefer disabling
|
|
|
234 them than releasing an unstable (but untested) binary package.
|
