Jim: Thanks, but I'm looking for something that can be used somewhat automatically -- the function in question would be user-provided and passed to my "chunking" algorithm, so in this case it would be the end-user (not me) who would have to embed these -- would Rprof(memory.profiling=TRUE) # my function Rprof(NULL) ... and then taking the max of the tseries output be a reasonable approach? If so, which of the three outputs (vsize.small vsize.large nodes) would be best compared against the available memory? Cheers! --j
On Thu, Jun 20, 2013 at 10:07 AM, jim holtman <jholtman at gmail.com> wrote:
What I would do is to use "memory.size()" to get the amount of memory being used. Do a call at the beginning of the function to determine the base, and then at other points in the code to see what the difference from the base is and keep track of the maximum difference. I am not sure if just getting the memory usage at the end would be sufficient since there may be some garbage collection in between, or you might be creating some large objects and then deleting/reusing them. So keep track after large chunks of code to see what is happening. On Thu, Jun 20, 2013 at 10:45 AM, Jonathan Greenberg <jgrn at illinois.edu> wrote:
Folks: I apologize for the cross-posting between r-help and r-sig-hpc, but I figured this question was relevant to both lists. I'm writing a function to be applied to an input dataset that will be broken up into chunks for memory management reasons and for parallel execution. I am trying to determine, for a given function, what the *maximum* memory usage during its execution is (which may not be the beginning or the end of the function, but somewhere in the middle), so I can "plan" for the chunk size (e.g. have a table of chunk size vs. max memory usage). Is there a trick for determining this? --j -- Jonathan A. Greenberg, PhD Assistant Professor Global Environmental Analysis and Remote Sensing (GEARS) Laboratory Department of Geography and Geographic Information Science University of Illinois at Urbana-Champaign 607 South Mathews Avenue, MC 150 Urbana, IL 61801 Phone: 217-300-1924 http://www.geog.illinois.edu/~jgrn/ AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
-- Jim Holtman Data Munger Guru What is the problem that you are trying to solve? Tell me what you want to do, not how you want to do it.
Jonathan A. Greenberg, PhD Assistant Professor Global Environmental Analysis and Remote Sensing (GEARS) Laboratory Department of Geography and Geographic Information Science University of Illinois at Urbana-Champaign 607 South Mathews Avenue, MC 150 Urbana, IL 61801 Phone: 217-300-1924 http://www.geog.illinois.edu/~jgrn/ AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007