# Parallel tasks with rake? **URL:** \ **Category:** Ruby\ **Created:** [November 7, 2006, 10:52am UTC](https://www.ruby-forum.com/t/parallel-tasks-with-rake/74669 "2006-11-07T10:52:02Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\ **Post date:** [November 7, 2006, 10:52am UTC](https://www.ruby-forum.com/t/parallel-tasks-with-rake/74669/1 "2006-11-07T10:52:02Z")
I was excited to switch my makefile system to rake, but I just noticed rake lacks a (GNU) make option I can’t live without: -j4. It’s like “make & ; make & ; make & ; make &” but with the parallel jobs properly synchronized. On a quad-core machine this makes a world of difference, being around four times faster than a simple ‘make’. As I understand it, rake should be able to determine which targets can be built in parallel just as (GNU) make does. rake’s multitask feature isn’t quite right because it shouldn’t be up to the user to determine parallel vs non-parallel tasks. In fact, the user should probably never decide parallelism by fiat; even if the user happens to be correct, it creates a maintenance problem whereby additional tasks will eventually render the parallel assertion false. Perhaps there is something I’m missing about ‘multitask’ ? ---
**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\ **Post date:** [November 7, 2006, 10:52am UTC](https://www.ruby-forum.com/t/parallel-tasks-with-rake/74669/2 "2006-11-07T10:52:06Z")
On 11/1/06, removed\_email\_address@domain.invalid [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > probably never decide parallelism by fiat; even if the user happens to > be correct, it creates a maintenance problem whereby additional tasks > will eventually render the parallel assertion false. > > Perhaps there is something I’m missing about ‘multitask’ ? How does Make figure out which jobs can be run in parallel? In my experience, it just runs everything in parallel, and things break when the tasks aren’t safely parallelizable. (e.g. doing make -j4 install on a FreeBSD port) ---
**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\ **Post date:** [November 7, 2006, 10:52am UTC](https://www.ruby-forum.com/t/parallel-tasks-with-rake/74669/3 "2006-11-07T10:52:12Z")
Wilson B. wrote: > > multitask feature isn’t quite right because it shouldn’t be up to > > the tasks aren’t safely parallelizable. (e.g. doing make -j4 install > > on a FreeBSD port) The Make program figures it out from the dependency graph. Unfortunately, the ‘install’ target you mention is a fake target which depends on nothing. Such uses of ‘install’ are common but incorrect. As a workaround, ‘install’ (and any other wrongly written targets) should be treated as a special case and should not be run with -j. ---
**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\ **Post date:** [November 7, 2006, 10:52am UTC](https://www.ruby-forum.com/t/parallel-tasks-with-rake/74669/4 "2006-11-07T10:52:17Z")
On Thu, 2 Nov 2006, removed\_email\_address@domain.invalid wrote: > depends on nothing. Such uses of ‘install’ are common but incorrect. > As a workaround, ‘install’ (and any other wrongly written targets) > should be treated as a special case and should not be run with -j. Would marking it as a .PHONY help? I see little advice in the GNU make manual about how to get this sort of thing right for parallel make. Where should one look for such info? ``` Hugh ``` ---
**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\ **Post date:** [November 7, 2006, 10:52am UTC](https://www.ruby-forum.com/t/parallel-tasks-with-rake/74669/5 "2006-11-07T10:52:49Z")
On Nov 1, 2006, at 1:15 PM, removed\_email\_address@domain.invalid wrote: > Rake is a very nice tool, but it needs a -j equivalent in order to be > considered as an alternative to make (at least in my case). I put it > out to enterprising rubyists to give rake genuine parallel task > execution. I bet the pieces of this could be written up as a fun Ruby Q. or two. Determining what can be done in parallel from the dependancy graph in particular sounds like a great quiz topic. Then you could Rakify the solutions and submit a patch. Just a thought… James Edward G. II ---
**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\ **Post date:** [November 7, 2006, 10:52am UTC](https://www.ruby-forum.com/t/parallel-tasks-with-rake/74669/6 "2006-11-07T10:52:46Z")
Hugh S. wrote: > Would marking it as a .PHONY help? I see little advice in the GNU > make manual about how to get this sort of thing right for parallel > make. Where should one look for such info? First of all, the workaround isn’t so bad: ``` $ make -j4 && make install ``` In order for ‘make -j4 install’ to work, the install target must simply depend on everything it needs. Needless to say, most Makefile writers assume a ‘make && make install’ invocation and don’t write proper prerequisites for the install target. In other words, they assume a single-process make. Marking the target .PHONY just means it will run unconditionally (so for example Make won’t be confused by a file named ‘install’) and won’t affect the correctness or incorrectness of the dependencies. To get back to the original point, the dependency graph is what decides parallelism, not the other way around. Unless I misunderstand the rake docs, the ‘multitask’ feature of rake is dangerous and should be removed (see my original post). Rake is a very nice tool, but it needs a -j equivalent in order to be considered as an alternative to make (at least in my case). I put it out to enterprising rubyists to give rake genuine parallel task execution. ---
**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\ **Post date:** [November 7, 2006, 10:53am UTC](https://www.ruby-forum.com/t/parallel-tasks-with-rake/74669/7 "2006-11-07T10:53:00Z")
removed\_email\_address@domain.invalid wrote: > To get back to the original point, the dependency graph is what > decides parallelism, not the other way around. In the current implementation, is the dependency graph explicitly built by rake? Or does it simply recursively run the dependencies as it goes along? Guillaume.