Skip to content

parallel::mclapply() dummy function on Windows?

9 messages · Tim Triche, Jr., Brian Ripley, Martin Morgan +1 more

Original
Brian Ripley
#
On Thu, 6 Oct 2011, Tim Triche, Jr. wrote:

            

      
      
      
    

        
Why would it make it easier?  And how could using a dummy for 'most 
users' (who are on Windows) offer them 'good parallel support'?

            

      
      
      
    

        
Take a look at e.g. package 'boot' to see how to offer alternatives. 
(A version that uses 'parallel' is pending on CRAN, or see 
http://www.stats.ox.ac.uk/pub/R/boot_1.3-3.tar.gz .) Package 
'parallel' may in future offer a higher-level abstraction layer that 
makes offers such a choice, but as the 'boot' code shows, deciding 
what to send to the workers in a snow-style cluster is not simple.

Note that it is not just Windows that lacks fork support: some 
front-ends (notably Rstudio) do not work with forking at present. And 
some parts of parallel (and multicore/snow) do not work reliably on 
some OSes (e.g. Solaris).

            

      
      
      
    

        
Please do follow the posting guide: no HTML and use the signature 
block for your real name and credentials.

            

      
      
      
    

        

      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        
      

    

    

      
      

        


  

        

      Brian Ripley
    

    

      
      #
    

  


  

      
On Fri, 7 Oct 2011, Tim Triche, Jr. wrote:

        

            

      
      
      
    

        
*I* don't: the author did.  It was the recommendation for S long ago: 
.S is taken and .q stood for qpe, at one time (mind 1980s, AFAIR) the 
proposed name for 'new S'.

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      John Fox
    

    

      
      #
    

  


  

      
Dear Tim,

            

      
      
      
    

        
On

            

      
      
      
    

        
users'

            

      
      
      
    

        
for

            

      
      
      
    

        
Unix.

            

      
      
      
    

        
parallelize

            

      
      
      
    

        
Windows.

            

      
      
      
    

        
not

            

      
      
      
    

        
Why don't you just apply the approach you initially suggested in your own
package, defining mclapply() the way you want it? 

I hope this helps,
 John

            

      
      
      
    

        
but

            

      
      
      
    

        
extension

            

      
      
      
    

        
flags

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Martin Morgan
    

    

      
      #
    

  


  

      
On 10/07/2011 06:03 PM, John Fox wrote:

            

      
      
      
    

        
Hi John et al.,

Individual packages will become littered with ad hoc solutions, 
constructed without, for instance, the wisdom and experience of Prof. 
Ripley about platforms or environments in which it is appropriate to use 
mclapply. For instance, Tim's pseudo-code if (Windows) ... translated as 
if (.Platform$OS.type == "windows") doesn't sound like its the correct 
test; at least

   exists("mclapply", getNamespace("parallel"))

but probably more. Also, doesn't parallel's name space differ between 
platforms, requiring the package author to import(parallel) rather than 
the better practice of importFrom(parallel, mclapply) ?

Martin

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      John Fox
    

    

      
      #
    

  


  

      
Dear Martin,

I don't have an opinion about whether what Tim wants to do is a good idea,
but was responding to his comment that he would need "parallel=FALSE flags
all over the place." Why could he not simply define

 	mclapply <- if (.Platform$OS.type == "windows") base::lapply else
parallel::mclapply

in his package?

Best,
 John

            

      
      
      
    

        
On

            

      
      
      
    

        
constructed

            

      
      
      
    

        
if

            

      
      
      
    

        
at

            

      
      
      
    

        
the

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Brian Ripley
    

    

      
      #
    

  


  

      
On Sat, 8 Oct 2011, John Fox wrote:

        

            

      
      
      
    

        
Because mclapply has additional arguments that would be passed by FUN 
to lapply as part of ... .

We are contemplating having wrappers of mclapply and pvec on Windows 
equivalent to the behaviour with mc.cores = 1 on Unix.  But that is 
nothing to do with original specious claim to which I responded: if 
you want good parallel performance for most users you need also to 
support both parLapply and mclapply (or at least, parLapply with a 
fork cluster).

I think the import issue is a red herring: these functions are not 
called often enough for parallel::mclapply to be inefficient.  And 
really importFrom is only better practice for things that will always 
be used, since it moves the computation from as-needed to every time 
the package is loaded.

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

      
      
    

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      John Fox
    

    

      
      #
    

  


  

      
Dear Brian,

            

      
      
      
    

        
I did think of that and took a look at mclapply() before I responded. All of
the additional arguments occur after ... and have defaults. I assumed from
the original posting that Tim Triche is using the defaults (otherwise I
don't think he would have made his original suggestion), but even if he is
not, he could define mclapply() in his package as something like

mclapply <- if (.Platform$OS.type != "windows") parallel::mclapply
	else function(X, FUN, ..., mc.preschedule = TRUE, mc.set.seed =
TRUE,
         			mc.silent = FALSE, mc.cores =
getOption("mc.cores", 2L),
         			mc.cleanup = TRUE, mc.allow.recursive =
TRUE))
			base::lapply(X, FUN, ...)

As I said, I won't pretend that I know whether his general approach is
sound.

Best,
 John

            

      
      
      
    

        
nothing

            

      
      
      
    

        
parLapply

            

      
      
      
    

        
importFrom

            

      
      
      
    

        
moves