GNU Make parallel build

2 minute read

Related: Cmake fundamentals


GNU Make is one of the most popular build systems of all time. make is nearly ubiquitously used to call GNU Make.

The simplest reasons to use Make include:

  • saving time on builds, only recompile parts of the code that changed
  • ability to build in parallel (big speedup)
  • avoid copy-paste mistakes with excessively long compile commands

Note: In general, don’t build on ExFAT drives, since if the build script tries to make symbolic (soft) links, it will fail since ExFAT doesn’t allow soft links.

GNU Make parallel builds

By default, GNU Make only uses one thread. For most programs, speeding up compilation is desirable and accomplished via the -j or --jobs option:

make -j

The -j option alone uses all physical + virtual CPU cores. On a typical laptop, 4 to 8 threads will thus be compiling simultaneously where possible.

Caveats

In certain special cases like older Raspberry Pi computers, the CPU may go into thermal cutback or suffer undervoltage spikes if using an inadequate power adapter. While the better solution is to provide adequate DC power and CPU cooling, one may choose to compromise by manually specifying the number of threads. For example, with the Raspberry Pi I often use 2 threads.

make -j2 -l 1

The automatic load-limiting -l 1 is described below.

Keeping system responsive

A possible downside of building large programs with make -j is that the computer can become non-responsive to input including mouse and keyboard. A symptom of this is that the Caps Lock keyboard light is very slow to respond. To mitigate the problem automatically, disallow a new thread to start if the system becomes too heavily loaded. This is accomplished with the -l or --max-load option.

Linux system load of 0 to 1 means approximately that no tasks are waiting. So if you want to keep your computer responsible to user input (you can continue using your laptop while big program builds in the background), consider something like -l 1. You can set the -l factor higher to allow make to have increasingly higher priority over other tasks, including user input.

On a big program, I might use:

make -j -l 1.5

trading off how sluggish I can browse the web or edit documents while building reasonably fast with Make.

Further background

The material below is optional, but useful to understand how to use Make even more effectively with automatically-generated Makefiles.

Cmake and Make

make uses Makefile(s), which are plain-text files that are simple to understand for simple cases of less than 10 lines, but quickly become challenging to manage effectively for non-specialists. A ready solution comes in the form of Makefile generators. CMake is among the most popular Makefile generators.

I use CMake on all projects involving compiled code. It’s important to use templates where reasonable to save time and help prevent silly mistakes. CMake comes with an integrated test suite called Ctest, which works very well with continuous integration systems for automated self test. Continuous integration should be a part of all software projects of all sizes, from 50 line programs to operating systems.

Notes

GNU Make parallel docs

Tags:

Categories:

Updated:

Leave a Comment