Is combining mclapply and gbm tasks possible using R-3.0.1 ?
I think the problem has more to do with the version of gbm that you're using, not the version of R. Looking over gbm 2.1 briefly, it looks like it always creates a cluster object and does the cross validation with parLapply, even when you specify n.cores = 1. That explains why you see the messages from your .Rprofile within the mclapply tasks. The error messages from socketConnection may be happening because you're creating four cluster objects on the same machine at about the same time, resulting in port collisions. If you had any control over how the cluster object was created by gbm, you might be able to specify different ports for the different mclapply tasks, but it doesn't look like gbm has made any provision for that kind of control. If this is important to you, I think you should either back off to an older version of gbm, or talk to the package maintainer. I don't think the package expects you to call gbm using mclapply. - Steve On Wed, Aug 21, 2013 at 5:45 AM, Patrick Connolly
<p_connolly at slingshot.co.nz> wrote:
Apologies for such a long question. The question is fairly simple but takes a lot of describing.
sessionInfo()
R version 3.0.1 (2013-05-16) Platform: x86_64-unknown-linux-gnu (64-bit) locale: [1] LC_CTYPE=en_NZ.UTF-8 LC_NUMERIC=C [3] LC_TIME=en_NZ.UTF-8 LC_COLLATE=en_NZ.UTF-8 [5] LC_MONETARY=en_NZ.UTF-8 LC_MESSAGES=en_NZ.UTF-8 [7] LC_PAPER=C LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_NZ.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] datasets parallel splines grDevices utils stats graphics [8] methods base other attached packages: [1] gbm_2.1 survival_2.37-4 cairoDevice_2.19 lattice_0.20-15 loaded via a namespace (and not attached): [1] grid_3.0.1 multicore_0.1-7 tools_3.0.1 Using a system with the above characteristics, I made a function modifying some of the code from the examples in the gbm() function help. The objective was to run some examples with different seeds. And to do those in parallel using mclapply. In the interests of limiting the size of this email body and of avoiding email software munging the function code, I've put the code into the attached file testing.fn.sc which can be sourced into an R session. That function runs fine when I use an unupdated installation of R-2.13.1 and gbm 1.6-3.1 being quite capable of using four cores simultaneously. (It needs slight modification to use multicore instead of parallel and the call to gbm has no n.cores parameter.)
testing(4)
2013-08-21 20:57:31 Begin using multicore method with phony data with 4 cores.
Core 1 uses 20442
Core 2 uses 20443
Core 3 uses 20445
Core 4 uses 20447
2013-08-21 20:57:36
....Completed testing multicore method with invented data.
$a
CV Test OOB
1 126 131 79
[...]
$d
CV Test OOB
1 123 140 81
However, when I try it with the current versions I get this:
system.time(bbb <- testing(4))
2013-08-21 16:18:03 Begin using multicore method with phony data with 4 cores.
Core 1 uses 22812
Core 2 uses 22814
Core 3 uses 22816
Core 4 uses 22819
This session PID is 22821:
begun at 2013-08-21 16:18:04:
This session PID is 22829:
begun at 2013-08-21 16:18:04:
This session PID is 22838:
begun at 2013-08-21 16:18:04:
This session PID is 22847:
begun at 2013-08-21 16:18:04:
2013-08-21 16:18:07
....Completed testing multicore method with invented data.
user system elapsed
0.460 1.760 3.926
Warning message:
In mclapply(subsets, FUN = test.gbm, mc.cores = nc, mc.cleanup = FALSE, :
3 function calls resulted in an error
bbb$b
[1] "Error in socketConnection(\"localhost\", port = port, server = TRUE, blocking = TRUE, : \n cannot open the connection\n"
attr(,"class")
[1] "try-error"
attr(,"condition")
<simpleError in socketConnection("localhost", port = port, server = TRUE, blocking = TRUE, open = "a+b", timeout = timeout): cannot open the connection>
(bbb$a works properly and the errors on bbb$c and bbb$d are identical to the above.) The two lines that look like this This session PID is 22847: begun at 2013-08-21 16:18:04: will look mysterious. It's explained by the fact that my .Rprofile cats the beginning time and the process id used at the beginning of each R session (handy to know that sometimes). Those outputs indicate that extra R processes are started, and I assume end with nothing to do. That happens even when there is no problem with mclapply such as when only a single core is used.
system.time(aaa <- testing())
2013-08-21 16:09:46 Begin using multicore method with phony data with 1 cores. Core 1 uses 18484 This session PID is 18487: begun at 2013-08-21 16:09:46: Core 2 uses 18511 This session PID is 18543: begun at 2013-08-21 16:09:50: Core 3 uses 18553 This session PID is 18556: begun at 2013-08-21 16:09:54: Core 4 uses 18609 This session PID is 18612: begun at 2013-08-21 16:09:58: 2013-08-21 16:10:01 ....Completed testing multicore method with invented data. user system elapsed 1.540 2.070 15.488
Running that same code (minus the PID stuff) on a Windows 7
installation on identical hardware runs in about half that time.
Since I don't know how to get the equivalent to the PID information on
Windows, I can't tell if extra R processes are started on that
platform too. However, running a more demanding task did seem to show
that more than one core was being used as though the OS is capable of
a degree of parallelling even when the R tasks are done serially.
My question is how can I use gbm and mclapply without reverting to an
ancient R version?
TIA
--
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.
___ Patrick Connolly
{~._.~} Great minds discuss ideas
_( Y )_ Average minds discuss events
(:_~*~_:) Small minds discuss people
(_)-(_) ..... Eleanor Roosevelt
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.
_______________________________________________ R-sig-hpc mailing list R-sig-hpc at r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-hpc