GNU Make parallel build

GNU Make is an important legacy build system. Ninja build system is recommended for new and upgrading projects–Ninja works with CMake and Meson among others.

In general, build systems are used to:

  • save time on builds, only recompile parts of the code that changed
  • build in parallel (big compile time speedup)
  • avoid copy-paste mistakes with excessively long compile commands or piles of Bash scripts

In general, don’t build on FAT or ExFAT drives, since if the build script tries to make symbolic (soft) links, it will fail since ExFAT doesn’t allow soft links.

GNU Make parallel builds

By default, GNU Make only uses one thread. For most programs, speeding up compilation is desirable and accomplished via the -j or --jobs option:

make -j

The -j option alone uses all virtual CPU cores. On a typical laptop, 4 to 8 threads will thus be compiling simultaneously where possible. On some systems such as Windows Subsystem for Linux, the bare make -j -l2 may overwhelm the computer because WSL1 doesn’t seem to correctly report the load factor. Thus to do parallel builds with GNU Make on WSL1, manually set the number of parallel build threads perhaps equal to or less than the number of physical cores in your PC. For example an 8 physical core laptop would compile with make -j8.

Caveats

In certain special cases like older Raspberry Pi computers, the CPU may go into thermal cutback or suffer undervoltage spikes if using an inadequate power adapter. While the better solution is to provide adequate DC power and CPU cooling, one may choose to compromise by manually specifying the number of threads. For example, with the Raspberry Pi I often use 1 or 2 threads.

make -j1

# or

make -j2

RAM limitation

make -j does not consider system RAM, so your computer may become overwhelmed. Try make -j2 or so until the system memory isn’t over-consumed.

Keeping system responsive

A possible downside of building large programs with make -j is that the computer can become non-responsive to input including mouse and keyboard. A symptom of this is that the Caps Lock keyboard light is very slow to respond. To mitigate the problem automatically, disallow a new thread to start if the system becomes too heavily loaded. This is accomplished with the -l or --max-load option.

Linux system load of 0 to 1 means approximately that no tasks are waiting. So if you want to keep your computer responsible to user input (you can continue using your laptop while big program builds in the background), consider something like -l 1. You can set the -l factor higher to allow make to have increasingly higher priority over other tasks, including user input.

Notes

Instead of writing Makefiles, we use Meson on all projects involving compiled code. Meson comes with an integrated parallel-executing test suite, which works very well with continuous integration systems for automated self test. Continuous integration should be a part of all software projects of all sizes, from 50 line programs to operating systems. We recommend Azure Pipelines or GitHub Actions for CI in general.

  • GNU Make parallel docs

Cmake fundamentals