parse_Rd and/or lazyload problem

Okay, then we both agree we should drop it.
Duncan Murdoch

No we don't. I can't provide a functioning mvbutils, or debug, until this is resolved.

I am trying to be a good citizen and prepare reproducible bug reports-- e.g. the 3 line example. It would be quicker for me to write some ugly hack that modifies base R and gets round the problem *for me*, but that doesn't seem the best outcome for R. A culture which discourages careful bug reporting is unhealthy culture.
Sorry.  What I thought you said was that you had spent several hours on 
it and didn't want to spend more time on it.  I've told you I don't want 
to work on it either.

If there is no way to trigger this bug without using internals, then it 
has not been demonstrated to be a bug in R.  It might be one, or it 
might be a bug in your code.  Often I'll work on things that are 
demonstrated bugs, but I won't commit several hours to debugging your code.

Duncan Murdoch
Mark Bravington

________________________________________
From: Duncan Murdoch [murdoch at stats.uwo.ca]
Sent: 02 November 2009 01:08
To: Bravington, Mark (CMIS, Hobart)
Cc: r-devel at r-project.org
Subject: Re: [Rd] parse_Rd and/or lazyload problem

On 31/10/2009 10:18 PM, Mark.Bravington at csiro.au wrote:
Does this happen in R-patched?  I've seen similar errors in 2.10.0, but
not in a current build.
Yes, still there in R-patched.

(Still haven't got to your code, this was in
mine.  I'm reluctant to spend time on code that is messing with
internals, because you might be using things in a way not intended by
the author.  Now, if you can show me some code that demonstrates the
problem without using internals directly, I'll follow up.)
I did try, but it's not completely possible, because 'makeLazyLoadDB' is internal and there is no public alternative (a pity-- it's useful). However, the problem(s) can be demonstrated without directly calling 'parse_Rd', and with 'lazyLoad' (public) instead of 'fetchRdDB' (private), as per "pointer 1" below. If you have a look at 'tools:::.install_package_Rd_objects', you'll see that my use of 'makeLazyLoadDB' is quite standard.

The problem is not easy to reproduce. It took 4-5 hours work to get the 3-line reproducible example that I posted, plus another couple since, so I'm also reluctant to spend more time...

Okay, then we both agree we should drop it.
Duncan Murdoch

The examples in my previous post still apply-- the first one involves just 3 statements-- but here are some more pointers I've unearthed since:

1. Sometimes 'fetchRdDB' or 'lazyLoad' called directly from the prompt doesn't work, but public 'Rd_db' (which directly calls 'fetchRdDB') does. I've experimented with copying the installed 'tools' package into a new library "d:/temp/fakelib", then stuff like this:

test> e <- new.env()
test> lazyLoad( 'd:/temp/fakelib/tools/help/tools', e) # original files tools.rdx, tools.rdb
test> e <- as.list( e) # force evaluation
test> tools:::makeLazyLoadDB( e, 'd:/temp/fakelib/tools/help/tools') # modify tools.rd*
test> e1 <- new.env()
test> lazyLoad( 'd:/temp/fakelib/tools/help/tools', e1)
test> as.list( e1) # try to force evaluation...
Error in as.list.environment(e1) :
  cannot allocate memory block of size 2.7 Gb
test>
test> Rd_db( 'tools', 'd:/temp/fakelib') # no probs !?

2. Sometimes 'fetchRdDB' or 'lazyLoad' will fail in one R session, but will work in a fresh session on exactly the same files. For example, after restarting R, the previous commands involving 'e1' work fine.

Mark

Duncan Murdoch

I'm encountering problems when making lazy-loadable databases of the output from 'parse_Rd'. The lazy-
load database is of seemingly limitless size when I try to reload it... Admittedly I am using functions
that I'm not really supposed to use, which is why this isn't a bug report, but there does seem to be
something strange going on; my code is very similar to code that lives inside
'tools:::.install_package_Rd_objects'. The problems occur with just-released R2.10.0 on Windows.
object.size() has problems when working on Rd objects, because it counts
every environment separately, even though they may all be references to
the same one. I haven't looked at your code, but that could be a problem.

Mark

________________________________________
From: Duncan Murdoch [murdoch at stats.uwo.ca]
Sent: 31 October 2009 08:59
To: Bravington, Mark (CMIS, Hobart)
Cc: r-devel at r-project.org
Subject: Re: [Rd] parse_Rd and/or lazyload problem

On 30/10/2009 12:50 AM, Mark.Bravington at csiro.au wrote:
I'm encountering problems when making lazy-loadable databases of the output from 'parse_Rd'. The lazy-load database is of seemingly limitless size when I try to reload it... Admittedly I am using functions that I'm not really supposed to use, which is why this isn't a bug report, but there does seem to be something strange going on; my code is very similar to code that lives inside 'tools:::.install_package_Rd_objects'. The problems occur with just-released R2.10.0 on Windows.
object.size() has problems when working on Rd objects, because it counts
every environment separately, even though they may all be references to
the same one. I haven't looked at your code, but that could be a problem.

Duncan Murdoch

The examples below use files which can be found at ftp://ftp.csiro.au/MarkBravington, but you'll obviously need to modify the paths. The file "scrunge.Rd" is just "Rdiff.Rd" from the 'tools' package. The file "fakepack.7z" should unzip to create a fake package with a DESCRIPTION file and a "man" directory that contains two Rd files.

Transcript of first example:
eglist <- list( scrunge=tools:::prepare_Rd(  'd:/temp/scrunge.Rd',
    defines=.Platform$OS.type, stages='install', warningCalls=FALSE))
tools:::makeLazyLoadDB( eglist, 'd:/temp/ll', compress=FALSE)
tools:::fetchRdDB( 'd:/temp/ll')
# Error: cannot allocate vector of size 1.4 Gb

The second example triggers an error with 'tools:::.install_package_Rd_objects' itself. It doesn't happen the first time, but frequently does after the Rd files have been changed. Transcript:

# Make sure d:/temp/help/ is empty, then
test> tools:::.install_package_Rd_objects( 'd:/temp/fakepack', 'd:/temp')
test> tools:::fetchRdDB( 'd:/temp/help/temp')
# All good. Next, I *removed* one of the two Rd files in "d:/temp/fakepack/man", ...
# ...deleted "d:/temp/help/temp*", and tried again
test> tools:::.install_package_Rd_objects( 'd:/temp/fakepack', 'd:/temp')
test> tools:::fetchRdDB( 'd:/temp/help/temp')
Warning: Reached total allocation of 1535Mb: see help(memory.size)
Warning: Reached total allocation of 1535Mb: see help(memory.size)
Warning: Reached total allocation of 1535Mb: see help(memory.size)
Warning: Reached total allocation of 1535Mb: see help(memory.size)
Error: cannot allocate vector of size 1.9 Gb
# Or on other occasions I get
Error: internal error -3 in R_decompress1

Mark Bravington
CSIRO CMIS
Hobart
Australia

--please do not edit the information below--
Version:
 platform = i386-pc-mingw32
 arch = i386
 os = mingw32
 system = i386, mingw32
 status =
 major = 2
 minor = 10.0
 year = 2009
 month = 10
 day = 26
 svn rev = 50208
 language = R
 version.string = R version 2.10.0 (2009-10-26)
Windows XP (build 2600) Service Pack 2
Locale:
LC_COLLATE=English_Australia.1252;LC_CTYPE=English_Australia.1252;LC_MONETARY=English_Australia.1252;LC_NUMERIC=C;LC_TIME=English_Australia.1252
Search Path:
 .GlobalEnv, ROOT, package:grDevices, package:ad, package:chstuff, package:handy2, package:tweedie, package:statmod, package:handy, package:debug, package:mvbutils, mvb.session.info, package:tools, package:tcltk, package:stats, package:graphics, package:utils, package:methods, Autoloads, package:base
______________________________________________
R-devel at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

parse_Rd and/or lazyload problem

Thread (10 messages)