Parallel builds of clang causing paging.

[dtarditi]

I've confirmed that the parallel builds for automated tests are causing test machines to page (issue #267). I have tracked the problem to the amount of build parallelism being used in the MSBuild (Visual Studio solution) for clang generated by CMake.

CMake generates MSBuild files that set the /MP flag for the Microsoft Visual C++ compiler. When the C++ compiler is applied to a long list of files, it will launch as many compiler processes as there are CPU cores on your machine. For example,

cl A.c B.c, C.c D.c

causes the Microsoft Visual C++ compiler to launch 4 processes (one per file), if your machine as least 4 processors available (see this blog article). You can limit this in msbuild by setting the CL_MPCount property using the /p option. For example,

msbuild /p:CL_MPCount=nnn

where nnn is the maximum number of processes to launch.

When you use the /m:nnn option to msbuild, it launches as many build processes as are specified by nnn. If you use omit nnn and use only /m, it launches as many build processes as there are CPU cores on your machine. In our automated scripts, we were setting nnn to be 1/4 the number of CPU cores. The end result is that at times a quadratic number of compiler processes were being launched. If p is the number of processors, p^2/4 compilations were being launched. It is known that you can accidentally mis-use build parallelism with Visual Studio when using cmake: see the comments section for this Kitware CMake blog post.

The clang build does other things besides invoke the C++ compiler. It invokes tools like tblgen and builds libraries. There is lots of parallelism available in a clang/LLVM build, and the build system is better situated to recognize it and take advantage of it than invocations of the compiler driver. At the same time, the generated build system is invoking the C++ compiler with long lists of file arguments, so ratcheting individual build nodes down to no parallelism seems like a bad idea.

We need to make a trade-off: it seems better to set the build system parallelism to be high, and limit the individual compiler node parallelism to a constant. Modern OSes are very good at time-slicing CPU time across processes. They are not so good at time-slicing physical memory across processes. When there is more demand for physical memory than is actually available, this leads to virtual memory thrashing. Our goal is to create high CPU utilization while limiting memory usage to physical memory.

I've found that limiting the number of parallel C compiler processes spawned by the C++ compiler driver to 4 to 6 and setting the build system parallelism to the number of cores seems to achieve this. This is given 1 GB physical memory/core (166 MByte to 256 MBytes of memory/C++ compiler process). For 6 processes, you can do this by adding the following options to MSBuild:

msbuild /p:CL_CPUCount=6 /m

If you have less memory / core, you'll need to reduce the amount of build parallelism accordingly. I would suggest reducing the number of parallel C compiler launches.

[dtarditi]

This work is complete. The wiki has been updated to have the information in this issue and the md documentation has been updated too.