Hi, Is there any reason why .Rbuildignore is not used before copying package files in R CMD build? For some of the packages I develop I have rather large directories with miscellaneous files for testing and other purposes. They are in my .Rbuildignore (and .gitignore) file, but that doesn't prevent R CMD build from trying to copy them on the build process. Having them copied either breaks the build completely because /tmp directory gets out of space, or just slows it down a lot. So I wonder if there is a specific reason for this behavior and whether it could be change or controlled by some parameter. There is some discussion in the context of pkgbuild package: https://github.com/r-lib/pkgbuild/issues/59 It provides a hackish workaround for that, which also does not work on Windows. Best, Alexey
Apply .Rbuildignore before copying files in R CMD build
7 messages · Alexey Sergushichev, Duncan Murdoch, Kevin Ushey +1 more
On 2024-08-28 5:59 p.m., Alexey Sergushichev wrote:
Hi, Is there any reason why .Rbuildignore is not used before copying package files in R CMD build? For some of the packages I develop I have rather large directories with miscellaneous files for testing and other purposes. They are in my .Rbuildignore (and .gitignore) file, but that doesn't prevent R CMD build from trying to copy them on the build process. Having them copied either breaks the build completely because /tmp directory gets out of space, or just slows it down a lot. So I wonder if there is a specific reason for this behavior and whether it could be change or controlled by some parameter. There is some discussion in the context of pkgbuild package: https://github.com/r-lib/pkgbuild/issues/59 It provides a hackish workaround for that, which also does not work on Windows.
I think the reason is simplicity. The build process can add, delete or modify files. You wouldn't want that to happen on the original source files, so R copies the files to a temporary location to run things. If it applied .Rbuildignore first, then important files for the build might not be available, and the build could fail. Having an R package that needs so much data that you can't fit two copies of it on your disk is a really unusual situation. I think it will have to be up to you to fix it (by increasing your temp space, or decreasing the size of some of those files, or something else). Duncan Murdoch
Hi Alexey, It's kind of gross, but if you need a local solution, you could put your own executable `cp` script on your PATH somewhere, and have that do something "smart" when it's invoked by R. One way to detect that would be to check if the `R_CMD` environment variable is set. For example: https://gist.github.com/kevinushey/2351194ba540627831fa2d58073c097a For posterity, this was filed some time ago on R's bug tracker at: https://bugs.r-project.org/show_bug.cgi?id=17549 Best, Kevin
On Wed, Aug 28, 2024 at 3:00?PM Alexey Sergushichev <alsergbox at gmail.com> wrote:
Hi, Is there any reason why .Rbuildignore is not used before copying package files in R CMD build? For some of the packages I develop I have rather large directories with miscellaneous files for testing and other purposes. They are in my .Rbuildignore (and .gitignore) file, but that doesn't prevent R CMD build from trying to copy them on the build process. Having them copied either breaks the build completely because /tmp directory gets out of space, or just slows it down a lot. So I wonder if there is a specific reason for this behavior and whether it could be change or controlled by some parameter. There is some discussion in the context of pkgbuild package: https://github.com/r-lib/pkgbuild/issues/59 It provides a hackish workaround for that, which also does not work on Windows. Best, Alexey [[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
I think the reason is simplicity. The build process can add, delete or
modify files. You wouldn't want that to happen on the original source files, so R copies the files to a temporary location to run things.
If it applied .Rbuildignore first, then important files for the build
might not be available, and the build could fail. Yeah, I guess that makes sense. Thanks, I didn't realize that.
Having an R package that needs so much data that you can't fit two
copies of it on your disk is a really unusual situation. I think it will have to be up to you to fix it (by increasing your temp space, or decreasing the size of some of those files, or something else). These files are not required, but help to run and test some things. But I guess I have to deal with that. -- Alexey On Wed, Aug 28, 2024 at 5:11?PM Duncan Murdoch <murdoch.duncan at gmail.com> wrote:
On 2024-08-28 5:59 p.m., Alexey Sergushichev wrote:
Hi, Is there any reason why .Rbuildignore is not used before copying package files in R CMD build? For some of the packages I develop I have rather large directories with miscellaneous files for testing and other purposes. They are in my .Rbuildignore (and .gitignore) file, but that doesn't prevent R CMD build from trying to copy them on the build process. Having them copied either breaks the build completely because /tmp directory gets out of space, or just slows it down a lot. So I wonder if there is a specific reason for this behavior and whether it could be change or controlled by some parameter. There is some discussion in the context of pkgbuild package: https://github.com/r-lib/pkgbuild/issues/59 It provides a hackish workaround for that, which also does not work on Windows.
I think the reason is simplicity. The build process can add, delete or modify files. You wouldn't want that to happen on the original source files, so R copies the files to a temporary location to run things. If it applied .Rbuildignore first, then important files for the build might not be available, and the build could fail. Having an R package that needs so much data that you can't fit two copies of it on your disk is a really unusual situation. I think it will have to be up to you to fix it (by increasing your temp space, or decreasing the size of some of those files, or something else). Duncan Murdoch
It's kind of gross, but if you need a local solution, you could put
your own executable `cp` script on your PATH somewhere, and have that do something "smart" when it's invoked by R. Hah! Currently I opted out to using pkgbuild functions and specified `Config/build/copy-method: link` in my DESCRIPTION file ( https://pkgbuild.r-lib.org/reference/build.html). It's also a bit weird, but at least easy to do now.
For posterity, this was filed some time ago on R's bug tracker at:
https://bugs.r-project.org/show_bug.cgi?id=17549 Thanks for the link! -- ALexey
On Wed, Aug 28, 2024 at 5:13?PM Kevin Ushey <kevinushey at gmail.com> wrote:
Hi Alexey, It's kind of gross, but if you need a local solution, you could put your own executable `cp` script on your PATH somewhere, and have that do something "smart" when it's invoked by R. One way to detect that would be to check if the `R_CMD` environment variable is set. For example: https://gist.github.com/kevinushey/2351194ba540627831fa2d58073c097a For posterity, this was filed some time ago on R's bug tracker at: https://bugs.r-project.org/show_bug.cgi?id=17549 Best, Kevin On Wed, Aug 28, 2024 at 3:00?PM Alexey Sergushichev <alsergbox at gmail.com> wrote:
Hi, Is there any reason why .Rbuildignore is not used before copying package files in R CMD build? For some of the packages I develop I have rather large directories with miscellaneous files for testing and other purposes. They are in my .Rbuildignore (and .gitignore) file, but that doesn't prevent R CMD build from trying to copy them on the build process. Having them copied either breaks the build completely because /tmp directory gets out of space, or just slows it down a lot. So I wonder if there is a specific reason for this behavior and whether it could be change or controlled by some parameter. There is some discussion in the context of pkgbuild package: https://github.com/r-lib/pkgbuild/issues/59 It provides a hackish workaround for that, which also does not work on Windows. Best, Alexey [[alternative HTML version deleted]]
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
On Thu, Aug 29, 2024 at 12:12?AM Duncan Murdoch
<murdoch.duncan at gmail.com> wrote:
[...]
I think the reason is simplicity. The build process can add, delete or modify files. You wouldn't want that to happen on the original source files, so R copies the files to a temporary location to run things. If it applied .Rbuildignore first, then important files for the build might not be available, and the build could fail.
AFAICT the ignored files are deleted right after the copy, so they are not present during the build process. (But FIXME.)
On 2024-08-29 3:34 p.m., G?bor Cs?rdi wrote:
On Thu, Aug 29, 2024 at 12:12?AM Duncan Murdoch <murdoch.duncan at gmail.com> wrote: [...]
I think the reason is simplicity. The build process can add, delete or modify files. You wouldn't want that to happen on the original source files, so R copies the files to a temporary location to run things. If it applied .Rbuildignore first, then important files for the build might not be available, and the build could fail.
AFAICT the ignored files are deleted right after the copy, so they are not present during the build process. (But FIXME.)
I think some builds do that, but builds of packages with vignettes generally do an install of the package, and that might need the ignored files. There could be other situations too. You probably know this, but for the benefit of those who don't: you can read the build operations in the 1100 line function tools:::.build_packages, which starts here: https://github.com/wch/r-source/blob/1bdf2503322b43ce8698008eb5bc1f55bc8a58c2/src/library/tools/R/build.R#L93 The prepare_pkg() function is run between the copy and the cleanup, and it might do a package install. Duncan Murdoch