Hi, all. I'm running R on a beowulf system, with software that can take a list of single-processor jobs and parcel them out over the system. (In case it's of interest, they're dual pentium 866 machines connected by fast ethernet.) The administrators of the system found that I was causing a huge load on the file server, and when they investigated it turned out that each process was reading in .RData 4 bytes at a time. Not kilobytes, bytes. The .RData in question is several megabytes, and more than one process is starting up at a time, so apparently it really adds up. I looked through the documentation, and a little bit in some of the source, and haven't been able to find how exactly .RData is read. Is this part of the program? Is it something I can set? Is it something strange happening with the file system on the beowulf cluster? Any help appreciated. Matt Wiener -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
loading .RData on distributed system
2 messages · Matthew Wiener, Luke Tierney
As far as I can tell from a quick look the file is opened with fopen and read with xdr. So unless the xdr implementation R is linked against is doing something very weird, the input should be buffered. On my RedHat 7.1 system strace output seems to suggest that the read is buffered with a buffer size of 4096. luke
On Fri, Jul 20, 2001 at 12:56:18PM -0400, Matthew Wiener wrote:
Hi, all. I'm running R on a beowulf system, with software that can take a list of single-processor jobs and parcel them out over the system. (In case it's of interest, they're dual pentium 866 machines connected by fast ethernet.) The administrators of the system found that I was causing a huge load on the file server, and when they investigated it turned out that each process was reading in .RData 4 bytes at a time. Not kilobytes, bytes. The .RData in question is several megabytes, and more than one process is starting up at a time, so apparently it really adds up. I looked through the documentation, and a little bit in some of the source, and haven't been able to find how exactly .RData is read. Is this part of the program? Is it something I can set? Is it something strange happening with the file system on the beowulf cluster? Any help appreciated. Matt Wiener -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
Luke Tierney University of Minnesota Phone: 612-625-7843 School of Statistics Fax: 612-624-8868 313 Ford Hall, 224 Church St. S.E. email: luke at stat.umn.edu Minneapolis, MN 55455 USA WWW: http://www.stat.umn.edu -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._