While I do enjoy Ben
& Jerry‘s Ice cream, here, I’m referring to setting up a distributed compilation cluster using simple machines all on the local subnet.
Introducing icecream, based on distcc, a wrapper to the compiler that allows making the best use of the spare processing power in your network.
I find myself doing a lot of compiling. I use gentoo so it comes with the territory. Speeding up the build process as much as possible is refreshing. As I have a lot of spare computing power at my disposal right now (but that’s a story for another blog post), it makes sense to utilize all of it.
Unfortunately networking processors at the moment is a complicated problem (although Tilera may have solved it). There are systems such as MPI, Mosix and Beowulf clusters, but it usually takes a lot of dedication and setup to get one working. Even when it does work, you still need software engineers who know the capabilities of the system to write software specifically.
My experience of using a Cluster of Workstations (CoW) has been quite disappointing since that usually involves defining some upper limit of resources that need to be allocated to your program. The programs tend to be written with these hardcoded values, number of processors, amount of ram, disk space etc.
What would be nice is for a cluster system that allows multiple heterogeneous computers (anyone can bring in any spare computer), load it with simple software/configuration that distributes processes and/or threads to the most available core. Preferably, with little or no modification to normal programs.
I don’t have the illusion that these problems have been satisfactorily resolved, but I present here an example that presents a solution to one of tasks that processing in parallel is good for.
The single threaded case
Maybe you have a slightly larger program, say, three or four source files and a Makefile. When you give the ‘make’ command, that will invoke ‘gcc -c’ for each source file and ‘gcc’ or ‘ld’ to link the intermediate object files into a final executable.
The nice bit here, is that each invocation of ‘gcc -c’ is only dependent on the compiler toolchain, not on other source files. We can get a speed-up by invoking the compiler to do these in parallel.
Single computer, Multiple processors
If you have a multicore system as found in almost any modern computer, you can reap the benefits of two or more compilations at the same time.
There is a little bit of an overhead, you can’t expect that but it gets close, and can save a significant amount of time if compiling tens, hundreds or even thousands of source files. This overhead can be reduced by specifying how many parallel jobs ‘make’ is to devise.
But we can do better than that.