Guys, I'm running R on both Windows & Linux. I'm looking at a number of packages for parallel execution. It seems that the most used packages are "snow" and "Rmpi". snow seems more user friendly, but it doesn't run on windows. I see from searching the mailing list that I'm not the first one to try it on Windows. There was a message that kind of shed some hope on the subject, but nothing else. Rmpi works well on windows (with DeinoMPI) but it's kind of low level, so before I embark in writing code similar to snow to have some high level constructs I though I'd ask here about other peoples experiences... Is anyone around here doing parallel R on Windows? If so could you share your experience? Cheers, // Giuseppe ************************************************************ HSBC Bank plc may be solicited in the course of its placement efforts for a new issue, by investment clients of the firm for whom the Bank as a firm already provides other services. It may equally decide to allocate to its own proprietary book or with an associate of HSBC Group. This represents a potential conflict of interest. HSBC Bank plc has internal arrangements designed to ensure that the firm would give unbiased and full advice to the corporate finance client about the valuation and pricing of the offering as well as internal systems, controls and procedures to identify and manage conflicts of interest. HSBC Bank plc Registered Office: 8 Canada Square, London E14 5HQ, United Kingdom Registered in England - Number 14259 Authorised and regulated by the Financial Services Authority. ************************************************************ ----------------------------------------- SAVE PAPER - THINK BEFORE YOU PRINT! This transmission has been issued by a member of the HSBC Group "HSBC" for the information of the addressee only and should not be reproduced and/or distributed to any other person. Each page attached hereto must be read in conjunction with any disclaimer which forms part of it. Unless otherwise stated, this transmission is neither an offer nor the solicitation of an offer to sell or purchase any investment. Its contents are based on information obtained from sources believed to be reliable but HSBC makes no representation and accepts no responsibility or liability as to its completeness or accuracy.
Snow or alternative MPI packages on Windows
6 messages · giuseppe1.milicia at hsbcib.com, Martin Morgan, Markus Schmidberger +1 more
giuseppe1.milicia at hsbcib.com writes:
Guys, I'm running R on both Windows & Linux. I'm looking at a number of packages for parallel execution. It seems that the most used packages are "snow" and "Rmpi". snow seems more user friendly, but it doesn't run on windows. I see from searching the mailing list that I'm not the first one to try it on Windows. There was a message that kind of shed some hope on the subject, but nothing else. Rmpi works well on windows (with DeinoMPI) but it's kind of low level, so before I embark in writing code similar to snow to have some high level constructs I though I'd ask here about other peoples experiences...
Rmpi has low-level facilities, but things like mpi.spawn.Rslaves, mpi.parLapply and mpi.bcast.cmd are high-level in the same way that snow is. Hopefully others will comment with windows-specific input. Martin
Is anyone around here doing parallel R on Windows? If so could you share your experience? Cheers, // Giuseppe ************************************************************ HSBC Bank plc may be solicited in the course of its placement efforts for a new issue, by investment clients of the firm for whom the Bank as a firm already provides other services. It may equally decide to allocate to its own proprietary book or with an associate of HSBC Group. This represents a potential conflict of interest. HSBC Bank plc has internal arrangements designed to ensure that the firm would give unbiased and full advice to the corporate finance client about the valuation and pricing of the offering as well as internal systems, controls and procedures to identify and manage conflicts of interest. HSBC Bank plc Registered Office: 8 Canada Square, London E14 5HQ, United Kingdom Registered in England - Number 14259 Authorised and regulated by the Financial Services Authority. ************************************************************ ----------------------------------------- SAVE PAPER - THINK BEFORE YOU PRINT! This transmission has been issued by a member of the HSBC Group "HSBC" for the information of the addressee only and should not be reproduced and/or distributed to any other person. Each page attached hereto must be read in conjunction with any disclaimer which forms part of it. Unless otherwise stated, this transmission is neither an offer nor the solicitation of an offer to sell or purchase any investment. Its contents are based on information obtained from sources believed to be reliable but HSBC makes no representation and accepts no responsibility or liability as to its completeness or accuracy.
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Martin Morgan Computational Biology / Fred Hutchinson Cancer Research Center 1100 Fairview Ave. N. PO Box 19024 Seattle, WA 98109 Location: Arnold Building M2 B169 Phone: (206) 667-2793
Hello, we are using Rmpi and snow on windows. It is working very well. We are using Windows Server 2003 and *MPICH2. I probably will manage to have some tests with the new snow (*snow_0.3-3)* version this week. Best Markus Martin Morgan schrieb:
giuseppe1.milicia at hsbcib.com writes:
Guys,
I'm running R on both Windows & Linux. I'm looking at a number of packages
for parallel execution. It seems that the most used packages are "snow" and
"Rmpi".
snow seems more user friendly, but it doesn't run on windows. I see from
searching the mailing list that I'm not the first one to try it on Windows.
There was a message that kind of shed some hope on the subject, but nothing
else.
Rmpi works well on windows (with DeinoMPI) but it's kind of low level, so
before I embark in writing code similar to snow to have some high level
constructs I though I'd ask here about other peoples experiences...
Rmpi has low-level facilities, but things like mpi.spawn.Rslaves, mpi.parLapply and mpi.bcast.cmd are high-level in the same way that snow is. Hopefully others will comment with windows-specific input. Martin
Is anyone around here doing parallel R on Windows? If so could you share your experience? Cheers, // Giuseppe ************************************************************ HSBC Bank plc may be solicited in the course of its placement efforts for a new issue, by investment clients of the firm for whom the Bank as a firm already provides other services. It may equally decide to allocate to its own proprietary book or with an associate of HSBC Group. This represents a potential conflict of interest. HSBC Bank plc has internal arrangements designed to ensure that the firm would give unbiased and full advice to the corporate finance client about the valuation and pricing of the offering as well as internal systems, controls and procedures to identify and manage conflicts of interest. HSBC Bank plc Registered Office: 8 Canada Square, London E14 5HQ, United Kingdom Registered in England - Number 14259 Authorised and regulated by the Financial Services Authority. ************************************************************ ----------------------------------------- SAVE PAPER - THINK BEFORE YOU PRINT! This transmission has been issued by a member of the HSBC Group "HSBC" for the information of the addressee only and should not be reproduced and/or distributed to any other person. Each page attached hereto must be read in conjunction with any disclaimer which forms part of it. Unless otherwise stated, this transmission is neither an offer nor the solicitation of an offer to sell or purchase any investment. Its contents are based on information obtained from sources believed to be reliable but HSBC makes no representation and accepts no responsibility or liability as to its completeness or accuracy.
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Dipl.-Tech. Math. Markus Schmidberger Ludwig-Maximilians-Universit?t M?nchen IBE - Institut f?r medizinische Informationsverarbeitung, Biometrie und Epidemiologie Marchioninistr. 15, D-81377 Muenchen URL: http://ibe.web.med.uni-muenchen.de Mail: Markus.Schmidberger [at] ibe.med.uni-muenchen.de Tel: +49 (089) 7095 - 4599
Markus,
That's interesting. I'm using snow_0.3-0 that's the version on CRAN. I see
that on the maintainer's website newer versions are available...
I just tried snow_0.3-3 and as you said it actually works fine!
Thanks a lot for the help, that's exactly what I was looking for!
// Giuseppe
Markus
Schmidberger
<schmidb at ibe.med. To
uni-muenchen.de> Martin Morgan <mtmorgan at fhcrc.org>,
Giuseppe1 MILICIA/IBEU/HSBC at HSBC
16/07/2008 15:44 cc
Mail Size: 7121 r-devel at r-project.org
Subject
Please respond to Re: [Rd] Snow or alternative MPI
schmidb at ibe.med.u packages on Windows
ni-muenchen.de Entity
Investment Banking Europe - IBEU
Hello,
we are using Rmpi and snow on windows. It is working very well.
We are using Windows Server 2003 and *MPICH2.
I probably will manage to have some tests with the new snow
(*snow_0.3-3)* version this week.
Best
Markus
Martin Morgan schrieb:
giuseppe1.milicia at hsbcib.com writes:
Guys, I'm running R on both Windows & Linux. I'm looking at a number of
packages
for parallel execution. It seems that the most used packages are "snow"
and
"Rmpi". snow seems more user friendly, but it doesn't run on windows. I see from searching the mailing list that I'm not the first one to try it on
Windows.
There was a message that kind of shed some hope on the subject, but
nothing
else. Rmpi works well on windows (with DeinoMPI) but it's kind of low level,
so
before I embark in writing code similar to snow to have some high level constructs I though I'd ask here about other peoples experiences...
Rmpi has low-level facilities, but things like mpi.spawn.Rslaves, mpi.parLapply and mpi.bcast.cmd are high-level in the same way that snow is. Hopefully others will comment with windows-specific input. Martin
Is anyone around here doing parallel R on Windows? If so could you share your experience? Cheers, // Giuseppe ************************************************************ HSBC Bank plc may be solicited in the course of its placement efforts
for a
new issue, by investment clients of the firm for whom the Bank as a firm already provides other services. It may equally decide to allocate to
its
own proprietary book or with an associate of HSBC Group. This represents
a
potential conflict of interest. HSBC Bank plc has internal arrangements designed to ensure that the firm would give unbiased and full advice to
the
corporate finance client about the valuation and pricing of the offering
as
well as internal systems, controls and procedures to identify and manage conflicts of interest. HSBC Bank plc Registered Office: 8 Canada Square, London E14 5HQ, United Kingdom Registered in England - Number 14259 Authorised and regulated by the Financial Services Authority. ************************************************************ ----------------------------------------- SAVE PAPER - THINK BEFORE YOU PRINT! This transmission has been issued by a member of the HSBC Group "HSBC" for the information of the addressee only and should not be reproduced and/or distributed to any other person. Each page attached hereto must be read in conjunction with any disclaimer which forms part of it. Unless otherwise stated, this transmission is neither an offer nor the solicitation of an offer to sell or purchase any investment. Its contents are based on information obtained from sources believed to be reliable but HSBC makes no representation and accepts no responsibility or liability as to its completeness or accuracy.
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
-- Dipl.-Tech. Math. Markus Schmidberger Ludwig-Maximilians-Universit?t M?nchen IBE - Institut f?r medizinische Informationsverarbeitung, Biometrie und Epidemiologie Marchioninistr. 15, D-81377 Muenchen URL: http://ibe.web.med.uni-muenchen.de Mail: Markus.Schmidberger [at] ibe.med.uni-muenchen.de Tel: +49 (089) 7095 - 4599 ************************************************************ HSBC Bank plc may be solicited in the course of its placement efforts for a new issue, by investment clients of the firm for whom the Bank as a firm already provides other services. It may equally decide to allocate to its own proprietary book or with an associate of HSBC Group. This represents a potential conflict of interest. HSBC Bank plc has internal arrangements designed to ensure that the firm would give unbiased and full advice to the corporate finance client about the valuation and pricing of the offering as well as internal systems, controls and procedures to identify and manage conflicts of interest. HSBC Bank plc Registered Office: 8 Canada Square, London E14 5HQ, United Kingdom Registered in England - Number 14259 Authorised and regulated by the Financial Services Authority. ************************************************************ ----------------------------------------- SAVE PAPER - THINK BEFORE YOU PRINT! This transmission has been issued by a member of the HSBC Group "HSBC" for the information of the addressee only and should not be reproduced and/or distributed to any other person. Each page attached hereto must be read in conjunction with any disclaimer which forms part of it. Unless otherwise stated, this transmission is neither an offer nor the solicitation of an offer to sell or purchase any investment. Its contents are based on information obtained from sources believed to be reliable but HSBC makes no representation and accepts no responsibility or liability as to its completeness or accuracy.
giuseppe1.milicia at hsbcib.com wrote on 07/16/2008 08:52 AM:
Guys, I'm running R on both Windows & Linux. I'm looking at a number of packages for parallel execution. It seems that the most used packages are "snow" and "Rmpi". snow seems more user friendly, but it doesn't run on windows. I see from searching the mailing list that I'm not the first one to try it on Windows. There was a message that kind of shed some hope on the subject, but nothing else. Rmpi works well on windows (with DeinoMPI) but it's kind of low level, so before I embark in writing code similar to snow to have some high level constructs I though I'd ask here about other peoples experiences... Is anyone around here doing parallel R on Windows? If so could you share your experience?
I remember working with Jack Dongarra's group back in the late '90's when Microsoft donated some money, hardware, and software to ensure that pvm, scalapack, and the like worked on Windows. My particular task was to port Atlas to Windows and benchmark installs with various compilers and commercial BLAS libraries. it was a total PITA, dealing with calling conventions between fortran and C/C++, and DLL hell. Anyway, the main researcher involved in the parallel Windows work realized (obviously) the hard problem was remotely connecting to Windows machines and spawning tasks, so he spent the majority of his time writing an rsh daemon, and it became kind of a stink because he went on to commercialize his work, buying the domain name myrshdaemon.com, and selling it. No one ever knew how much money he made. Point being, around ten years have passed and including Windows in a parallel setup is still a problem?!? Sheesh! Jeff
:)
Actually it turns out that DeinoMPI & Rmpi & the new version of snow do the
trick.
Jay Emerson suggested NetWorkSpaces which looks quite neat. For the time
being I'll probably stick to snow/Rmpi though.
So it seems that progress has been made!
// Giuseppe
Jeffrey Horner
<jeff.horner at vand
erbilt.edu> To
Giuseppe1 MILICIA/IBEU/HSBC at HSBC
17/07/2008 16:44 cc
Mail Size: 5648 r-devel at r-project.org
Subject
Re: [Rd] Snow or alternative MPI
packages on Windows
Entity
Investment Banking Europe - IBEU
giuseppe1.milicia at hsbcib.com wrote on 07/16/2008 08:52 AM:
Guys, I'm running R on both Windows & Linux. I'm looking at a number of
packages
for parallel execution. It seems that the most used packages are "snow"
and
"Rmpi". snow seems more user friendly, but it doesn't run on windows. I see from searching the mailing list that I'm not the first one to try it on
Windows.
There was a message that kind of shed some hope on the subject, but
nothing
else. Rmpi works well on windows (with DeinoMPI) but it's kind of low level, so before I embark in writing code similar to snow to have some high level constructs I though I'd ask here about other peoples experiences... Is anyone around here doing parallel R on Windows? If so could you share your experience?
I remember working with Jack Dongarra's group back in the late '90's when Microsoft donated some money, hardware, and software to ensure that pvm, scalapack, and the like worked on Windows. My particular task was to port Atlas to Windows and benchmark installs with various compilers and commercial BLAS libraries. it was a total PITA, dealing with calling conventions between fortran and C/C++, and DLL hell. Anyway, the main researcher involved in the parallel Windows work realized (obviously) the hard problem was remotely connecting to Windows machines and spawning tasks, so he spent the majority of his time writing an rsh daemon, and it became kind of a stink because he went on to commercialize his work, buying the domain name myrshdaemon.com, and selling it. No one ever knew how much money he made. Point being, around ten years have passed and including Windows in a parallel setup is still a problem?!? Sheesh! Jeff -- http://biostat.mc.vanderbilt.edu/JeffreyHorner ************************************************************ HSBC Bank plc may be solicited in the course of its placement efforts for a new issue, by investment clients of the firm for whom the Bank as a firm already provides other services. It may equally decide to allocate to its own proprietary book or with an associate of HSBC Group. This represents a potential conflict of interest. HSBC Bank plc has internal arrangements designed to ensure that the firm would give unbiased and full advice to the corporate finance client about the valuation and pricing of the offering as well as internal systems, controls and procedures to identify and manage conflicts of interest. HSBC Bank plc Registered Office: 8 Canada Square, London E14 5HQ, United Kingdom Registered in England - Number 14259 Authorised and regulated by the Financial Services Authority. ************************************************************ ----------------------------------------- SAVE PAPER - THINK BEFORE YOU PRINT! This transmission has been issued by a member of the HSBC Group "HSBC" for the information of the addressee only and should not be reproduced and/or distributed to any other person. Each page attached hereto must be read in conjunction with any disclaimer which forms part of it. Unless otherwise stated, this transmission is neither an offer nor the solicitation of an offer to sell or purchase any investment. Its contents are based on information obtained from sources believed to be reliable but HSBC makes no representation and accepts no responsibility or liability as to its completeness or accuracy.