multicore package: collecting results
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 06/29/2011 02:34 PM, Vincent Aubanel wrote:
Hi all, I'm using mclapply() of the multicore package for processing chunks of data in parallel --and it works great. But when I want to collect all processed elements of the returned list into one big data frame it takes ages. The elements are all data frames having identical column names, and I'm using a simple rbind() inside a loop to do that. But I guess it makes some expensive checking computations at each iteration as it gets slower and slower as it goes. Writing out to disk individual files, concatenating with the system and reading back from disk the resulting file is actually faster...
Why do you have to write to disk? Can you collect the results as a list L and then do.call(rbind,L) in one go?
Is there a magic argument to rbind() that I'm missing, or is there any other solution to collect the results of parallel processing efficiently? Thanks, Vincent
_______________________________________________ R-SIG-Mac mailing list R-SIG-Mac at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-mac
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4Lc1sACgkQc5UpGjwzenMofgCdFEzN6qO8e8owV9GCJ6Mwafvn n0oAoIZt5txoRS9Ma73XefnEzBDW19lE =BDfd -----END PGP SIGNATURE-----