|
150
|
1 # Libc mem* benchmarks
|
|
|
2
|
|
|
3 This framework has been designed to evaluate and compare relative performance of
|
|
|
4 memory function implementations on a particular host.
|
|
|
5
|
|
|
6 It will also be use to track implementations performances over time.
|
|
|
7
|
|
|
8 ## Quick start
|
|
|
9
|
|
|
10 ### Setup
|
|
|
11
|
|
|
12 **Python 2** [being deprecated](https://www.python.org/doc/sunset-python-2/) it is
|
|
|
13 advised to used **Python 3**.
|
|
|
14
|
|
|
15 Then make sure to have `matplotlib`, `scipy` and `numpy` setup correctly:
|
|
|
16
|
|
|
17 ```shell
|
|
|
18 apt-get install python3-pip
|
|
|
19 pip3 install matplotlib scipy numpy
|
|
|
20 ```
|
|
|
21
|
|
|
22 To get good reproducibility it is important to make sure that the system runs in
|
|
|
23 `performance` mode. This is achieved by running:
|
|
|
24
|
|
|
25 ```shell
|
|
|
26 cpupower frequency-set --governor performance
|
|
|
27 ```
|
|
|
28
|
|
|
29 ### Run and display `memcpy` benchmark
|
|
|
30
|
|
|
31 The following commands will run the benchmark and display a 95 percentile
|
|
|
32 confidence interval curve of **time per copied bytes**. It also features **host
|
|
|
33 informations** and **benchmarking configuration**.
|
|
|
34
|
|
|
35 ```shell
|
|
|
36 cd llvm-project
|
|
|
37 cmake -B/tmp/build -Sllvm -DLLVM_ENABLE_PROJECTS=libc -DCMAKE_BUILD_TYPE=Release
|
|
|
38 make -C /tmp/build -j display-libc-memcpy-benchmark-small
|
|
|
39 ```
|
|
|
40
|
|
|
41 ## Benchmarking regimes
|
|
|
42
|
|
|
43 Using a profiler to observe size distributions for calls into libc functions, it
|
|
|
44 was found most operations act on a small number of bytes.
|
|
|
45
|
|
|
46 Function | % of calls with size ≤ 128 | % of calls with size ≤ 1024
|
|
|
47 ------------------ | --------------------------: | ---------------------------:
|
|
|
48 memcpy | 96% | 99%
|
|
|
49 memset | 91% | 99.9%
|
|
|
50 memcmp<sup>1</sup> | 99.5% | ~100%
|
|
|
51
|
|
|
52 Benchmarking configurations come in two flavors:
|
|
|
53
|
|
|
54 - [small](libc/utils/benchmarks/configuration_small.json)
|
|
|
55 - Exercises sizes up to `1KiB`, representative of normal usage
|
|
|
56 - The data is kept in the `L1` cache to prevent measuring the memory
|
|
|
57 subsystem
|
|
|
58 - [big](libc/utils/benchmarks/configuration_big.json)
|
|
|
59 - Exercises sizes up to `32MiB` to test large operations
|
|
|
60 - Caching effects can show up here which prevents comparing different hosts
|
|
|
61
|
|
|
62 _<sup>1</sup> - The size refers to the size of the buffers to compare and not
|
|
|
63 the number of bytes until the first difference._
|
|
|
64
|
|
|
65 ## Benchmarking targets
|
|
|
66
|
|
|
67 The benchmarking process occurs in two steps:
|
|
|
68
|
|
|
69 1. Benchmark the functions and produce a `json` file
|
|
|
70 2. Display (or renders) the `json` file
|
|
|
71
|
|
|
72 Targets are of the form `<action>-libc-<function>-benchmark-<configuration>`
|
|
|
73
|
|
|
74 - `action` is one of :
|
|
|
75 - `run`, runs the benchmark and writes the `json` file
|
|
|
76 - `display`, displays the graph on screen
|
|
|
77 - `render`, renders the graph on disk as a `png` file
|
|
|
78 - `function` is one of : `memcpy`, `memcmp`, `memset`
|
|
|
79 - `configuration` is one of : `small`, `big`
|
|
|
80
|
|
|
81 ## Superposing curves
|
|
|
82
|
|
|
83 It is possible to **merge** several `json` files into a single graph. This is
|
|
|
84 useful to **compare** implementations.
|
|
|
85
|
|
|
86 In the following example we superpose the curves for `memcpy`, `memset` and
|
|
|
87 `memcmp`:
|
|
|
88
|
|
|
89 ```shell
|
|
|
90 > make -C /tmp/build run-libc-memcpy-benchmark-small run-libc-memcmp-benchmark-small run-libc-memset-benchmark-small
|
|
|
91 > python libc/utils/benchmarks/render.py3 /tmp/last-libc-memcpy-benchmark-small.json /tmp/last-libc-memcmp-benchmark-small.json /tmp/last-libc-memset-benchmark-small.json
|
|
|
92 ```
|
|
|
93
|
|
|
94 ## Useful `render.py3` flags
|
|
|
95
|
|
|
96 - To save the produced graph `--output=/tmp/benchmark_curve.png`.
|
|
|
97 - To prevent the graph from appearing on the screen `--headless`.
|
|
|
98
|
|
|
99
|
|
|
100 ## Under the hood
|
|
|
101
|
|
|
102 To learn more about the design decisions behind the benchmarking framework,
|
|
|
103 have a look at the [RATIONALE.md](RATIONALE.md) file.
|
