On Aug 12, 2017, at 4:10 PM, luke-tierney at uiowa.edu wrote:
On Sat, 12 Aug 2017, Dmitriy Selivanov wrote:
Strange because in my all my experiments calling malloc.trim always helped
- memory reported by top decreased to the level it supposed to be. Do you
have in mind case when calling malloc.trim won't do anything? Also
shouldn't MALLOC_TRIM_THRESHOLD_ env variable has impact on malloc.trim
calls? At the moment seems any value is ignored...
This is a question for glibc developers. For that matter, this entire
thread is really about tuning of the malloc in glibc and should
ideally be addressed upstream.
There has been some discussion of this in other contexts, e.g. Python at
https://nuald.blogspot.com/2013/06/memory-reclaiming-in-python.html
and emacs at
http://notes.secretsauce.net/notes/2016/04/08_glibc-malloc-inefficiency.html
As the Python posts poitns out, it is possible to use alternate malloc
implementations, either rebuilding R to use them or using LD_PRELOAD.
On Ubuntu for example, you can have R use jemalloc with
sudo apt-get install libjemalloc1
env LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 R
This does not seem to hold onto memory to the same degree, but I don't
know about any other aspect of its performance.
The emacs post suggests that calling malloc_trim may have more of an
effect in some cases: the post describes callign it via gdb on all
running processes and sees a substantial memory footprint drop for
emacs, Xorg and opera. I tried the same experiment on my system with
emacs, Xorg and firefox and didn't see what that post saw -- maybe 1
few percent recovery.
At this point I'm not sure we have enough data to show that adding
malloc_trim calls at the end of each GC, say, would warrant the
nusance of having to add configure checks. It would also be necessary
to make sure that adding this doesn't significantly ompact malloc
performance.
I don't know if this issue exists on Windows as well; it might as the
basic malloc we use there is the same as used in glibc (Dog Lea's
malloc).
Best,
luke
12 ???. 2017 ?. 6:09 ?? ???????????? "Simon Urbanek" <
simon.urbanek at r-project.org> ???????:
On Aug 11, 2017, at 12:57 PM, I?aki ?car <i.ucar86 at gmail.com> wrote:
2017-08-11 16:00 GMT+02:00 Martin Maechler <maechler at stat.math.ethz.ch>:
Dmitriy Selivanov <selivanov.dmitriy at gmail.com>
on Fri, 11 Aug 2017 17:33:31 +0400 writes:
Hi mailing list and R-core. Could someone from R-core please help me to
create account in bugzilla? I would like to submit issue related to
I will create one.
Your previous e-mails left me pretty clueless about what the
problem is that you want to solve ... but maybe others
understand better what you mean.
Note that in the case of such a relatively sophisticated wish
without a clear sign of a problem (in my view)
chances are not high that anything will change, unless someone
provides a (small footprint) patch towards the (R-devel aka
"trunk") sources *and* reproducible R code that depicts the
problem.
How to reproduce it:
a <- replicate(2e6, new.env()) # ~ 1.4 GB of memory
rm(a)
gc() # the R process still has the memory assigned
Right, but that's unavoidable because of the way Linux allocates memory -
see FAQ 7.42
The memory is free, Linux just keeps it for future allocations.
Running malloc.trim doesn't help, because the issue is fragmentation due
to the linear design of the pool - you likely will have another object on
top so in most practical cases malloc.trim() doesn't actually do anything.
You can always call malloc.trim() yourself is you think it helps, but it
doesn't in the general case. The only way to address that would be to move
allocated objects from top of the pool down, but that's not something R can
allow, because it cannot know which code still has SEXP pointers referring
to that object.
Cheers,
Simon
[[alternative HTML version deleted]]