Skip to content

update.packages() as ordinary user, /usr/lib/R/site-library is not writable

9 messages · Dirk Eddelbuettel, Chris Evans, Johannes Ranke

Original
Chris Evans
# 
I am moving from windoze on a Dell laptop to Debian but I seem to have 
hit a snag for R.  I managed to find the information to point an 
/etc/apt/sources.list entry at my local CRAN repository and have 
installed R 2.13.1 for squeeze from the Bristol UK repository.

I  installed a number of the additional R packages using synaptic which 
reported no errors but when I run my first update.packages() as an 
ordinary user I get this:


The downloaded packages are in
	?/tmp/Rtmp9LYziU/downloaded_packages?
Warning in install.packages(update[instlib == l, "Package"], l, 
contriburl = contriburl,  :
   'lib = "/usr/lib/R/library"' is not writable
Error in install.packages(update[instlib == l, "Package"], l, contriburl 
= contriburl,  :
   unable to install packages
In addition: Warning messages:
1: In install.packages(update[instlib == l, "Package"], l, contriburl = 
contriburl,  :
   installation of package 'rgl' had non-zero exit status
2: In install.packages(update[instlib == l, "Package"], l, contriburl = 
contriburl,  :
   installation of package 'XML' had non-zero exit status

Although all the compilations have run through, presumably with errors 
for rgl and XML, clearly the "not writable" status of /usr/lib/R/library 
has genuinely prevented the update.

I thought I'd seen strong advice here recently against updating 
libraries as root so I'm intrigued about this.  Can someone advise me 
what I should do?

Thanks in advance and huge thanks to all who do the porting to Debian 
and generally make R so good.

Chris
Dirk Eddelbuettel
# 
Welcome!
On 20 September 2011 at 21:20, Chris Evans wrote:
| I am moving from windoze on a Dell laptop to Debian but I seem to have 
| hit a snag for R.  I managed to find the information to point an 
| /etc/apt/sources.list entry at my local CRAN repository and have 
| installed R 2.13.1 for squeeze from the Bristol UK repository.

Good so far.
 
| I  installed a number of the additional R packages using synaptic which 
| reported no errors but when I run my first update.packages() as an 
| ordinary user I get this:

*Every* interaction with the package management system in Debian or Ubuntu
has to happen as root.  So do NOT say 'synaptic', say 'sudo synaptic'.
 
| The downloaded packages are in
| 	?/tmp/Rtmp9LYziU/downloaded_packages?
| Warning in install.packages(update[instlib == l, "Package"], l, 
| contriburl = contriburl,  :
|    'lib = "/usr/lib/R/library"' is not writable
| Error in install.packages(update[instlib == l, "Package"], l, contriburl 
| = contriburl,  :
|    unable to install packages
| In addition: Warning messages:
| 1: In install.packages(update[instlib == l, "Package"], l, contriburl = 
| contriburl,  :
|    installation of package 'rgl' had non-zero exit status
| 2: In install.packages(update[instlib == l, "Package"], l, contriburl = 
| contriburl,  :
|    installation of package 'XML' had non-zero exit status

If you get this, and think you used synaptic, you're confused.

Synaptic gets you prebuilt binaries from the repos it knows about.

What you show here is local compilation, possibly triggered by R. Not at all
the same.
 
| Although all the compilations have run through, presumably with errors 
| for rgl and XML, clearly the "not writable" status of /usr/lib/R/library 

Both r-cran-rgl and r-cran-xml exists _as pre-built binaries_. See above, do
not build them locally unless you have to AND know how to.

| has genuinely prevented the update.
| 
| I thought I'd seen strong advice here recently against updating 
| libraries as root so I'm intrigued about this.  Can someone advise me 
| what I should do?
| 
| Thanks in advance and huge thanks to all who do the porting to Debian 
| and generally make R so good.

It works actually really well indeed, but it is still on Linux / Unix --
which is a whole new world onto itself for people just joining.  As the joke
goes: "Unix is user-friendly, but it is picky about its friends."

Is there a local user group or a colleague you could bug locally?

Dirk
 
| Chris
| 
| -- 
| Chris Evans <chris at psyctc.org> Skype: chris-psyctc
| Consultant Psychiatrist in Psychotherapy, Notts. PDD network;
| Professor, Psychotherapy, Nottingham University
| *If I am writing from one of those roles, it will be clear. Otherwise*
| *my views are my own and not representative of those institutions    *
| If you have difficulty Emailing me on this address or getting a reply,
| send again but cc to:       chris dot evans at nottshc dot nhs dot uk
| and to:                     c dot evans at nottingham dot ac dot uk
| 
| _______________________________________________
| R-SIG-Debian mailing list
| R-SIG-Debian at r-project.org
| https://stat.ethz.ch/mailman/listinfo/r-sig-debian
Dirk Eddelbuettel
# 
Also, as an aside, R should not have tried to write to
/usr/lib/R/site-library.  Via the file /etc/R/Renviron, we define the library
path.  You should see this in R:

R> .libPaths()
[1] "/usr/local/lib/R/site-library" "/usr/lib/R/site-library"       "/usr/lib/R/library"           
R> 

and /usr/local/lib/R/site-library as the 1st directory in this path should be
the one being written too.   You can then make that very directory writeable
by you (my preference) or run the install step as root in R (more clumsy in
my book).

Dirk
Johannes Ranke
# 
Dear Chris, Dirk, dear list,

let me add some thoughts here:

Zitat von Dirk Eddelbuettel <edd at debian.org>:
So you are using the R binary packages I am maintaining for CRAN.
As Dirk pointed out, updating packages from within R under Linux/Unix  
requires a bit more than under Windows. However it is not a black art.  
Basically you need to have r-base-dev installed plus development  
packages for any libraries your R package is linking against.
@Dirk: He said he used update.packages() as an ordinary user.
It seems to me that Chris has r-cran-rgl and r-cran-xml from Debian  
squeeze, and R 2.13.1 from CRAN.
I agree that I would not recommend upgrading them if there is no good  
reason for doing so.
If you really want to upgrade R packages like rgl and XML that are  
available in Debian, I would recommend to do it the Debian way. This  
means adding the unstable sources to your /etc/apt/sources.list,  
running as root (using aptitude/apt as I am not familiar with synaptic)

     aptitude build-dep r-cran-rgl r-cran-xml

which gets build dependencies, then as a normal user go to some  
location where you want to build, i.e. ~/tmp and

     apt-get source -b r-cran-rgl r-cran-xml

You will find, if all goes well, updated .deb packages in your working  
directory that you can then install (again as root) e.g. using dpkg -i.

In this way you do not mix the packaging systems of Debian and R.

If you really want to do regular upgrades of all packages from within  
R (even though you decided to use Debian stable), you can get the  
build dependencies as above, but I believe Dirk would agree it would  
be better to remove the Debian packages first and then install R  
packages from within R in the first place.

Kind regards,

Johannes
Chris Evans
# 
Dirk Eddelbuettel sent the following at 20/09/11 21:37:
Thanks!
I ran synaptic from the Debian application menu which defaults to root 
after askling for the root password the first time I invoke it after a 
reboot. (I probably should have said that I've been using Debian for my 
internet server work for about 15 years: I'm not a Debian/linux newbie, 
just someone finally throwing away the reins from M$ to have a laptop 
running Debian as well!)
Perhaps but not how you think (I think!: just not clear in what I said.

I get that from invoking update.packages() in an R session launched from 
a user (chris) terminal session. Before that I had done an install of 
2.13.1 from synaptic calling r-cran-base and then finding R worked and 
then running synaptic again and getting those additional debs I wanted 
from the ones you packaged and the Bristol CRAN repository provided 
(r-cran-abind, r-cran-amelia, and others).
Agree and sorry I wasn't clear.
I wouldn't dream of compiling locally anything I can get from a deb you 
rolled!!
I like that phrase, hadn't heard it: nice. I do try to adapt to its 
demands and be a good friend and in 15 years have virtually never found 
it truly whimsical, it's usaully got a reason it doesn't like you even 
if it doesn't always tell you clearly what you did to offend!
Good question, in 15 years I haven't needed that but if someone on the 
list knows of them I could explore that.

However, I come back to feeling that there must be three separate 
problems here that come from within the packages I added after the 
initial r-cran-base install. My hunch is that there are things with XML 
and rgl that I will be able to track down when restrict to just updating 
them but that at the moment there is a more global problem which I'm 
guessing is caused by one package overriding the order of library path 
options seen in .libPaths() which starts with a perfectly writable local 
user directory and I am guessing there is some way that package has 
insisted that the install tries /usr/lib/R/library.

I don't know enough about the way that R and linux work agree the use of 
those library location options. From ?.libPaths and a bit more hunting 
around the R help I had the impression that the .libPath options would 
be used in order. I think I can remove the /usr ones by hacking in 
/etc/R but that presumably risks losing access to the main libraries 
that will have been installed there so that doesn't look a good step 
forward except perhaps to debug things.

I suspect that I could do a complete remove of all of R, start again 
installing r-cran-base, check that R installed by that updates fine run 
by a user and then I could use synaptic to add the additional deb 
packages you provide but doing that one by one and running R and 
update.packages() after each one to see if I don't hit this until afetr 
a particular deb but that's going to take quite a long time so I 
wondered if you or the list had quicker suggestions to debug this.

Sorry for length but hope that's clearer (got to try to be an acceptable 
friend on R lists as well as with linux and I really, really do 
appreciate the generosity of this list).

Chris
Dirk Eddelbuettel
# 
Hi Chris,

Thanks for patient follow-up even after I (as Johannes noted) evidently
overlooked key parts of your first email.
On 21 September 2011 at 07:49, Chris Evans wrote:
| I ran synaptic from the Debian application menu which defaults to root 
| after askling for the root password the first time I invoke it after a 
| reboot. (I probably should have said that I've been using Debian for my 
| internet server work for about 15 years: I'm not a Debian/linux newbie, 
| just someone finally throwing away the reins from M$ to have a laptop 
| running Debian as well!)

Perfect.

| I get that from invoking update.packages() in an R session launched from 
| a user (chris) terminal session. Before that I had done an install of 

I do that too, as well as calling the 'install.r' helper script from littler
(in its examples/ diretory, if interested).

This requires that you make /usr/local/lib/R/site-library/ writable for _you_
as user chris, else you cannot.

[ It also assumes that the Renviron setting mentioned last time is still
valid, do this:

edd at max:~$ R -q -e 'print(.libPaths())'
R> print(.libPaths())
[1] "/usr/local/lib/R/site-library" "/usr/lib/R/site-library"       "/usr/lib/R/library"           
R> 
R> 

It _must_ be first as here.  ]

Now, /usr/local/lib/R/site-library/ comes shipped from the r-base-core
package as owned by group 'staff' so one solution is to add yourself to group
staff:

edd at max:~$ ls -ld /usr/local/lib/R/site-library/
drwxrwsr-x 136 root staff 4096 2011-09-20 07:44 /usr/local/lib/R/site-library/
edd at max:~$ id
uid=1000(edd) gid=1000(edd)
groups=1000(edd),4(adm),20(dialout),24(cdrom),27(sudo),44(video),46(plugdev),50(staff),[.....]
edd at max:~$ 

If those two conditions are met, you should be able to run install.packages()
as you.  I still prefer it from the shell as I sometimes need another apt-get
call for a missing library etc.

| I wouldn't dream of compiling locally anything I can get from a deb you 
| rolled!!

:-)

| However, I come back to feeling that there must be three separate 
| problems here that come from within the packages I added after the 
| initial r-cran-base install. My hunch is that there are things with XML 
| and rgl that I will be able to track down when restrict to just updating 

"I would not dream of compiling locally" either.  You could start by locally
re-building the packages.  Which requires the build depends.

| them but that at the moment there is a more global problem which I'm 
| guessing is caused by one package overriding the order of library path 
| options seen in .libPaths() which starts with a perfectly writable local 
| user directory and I am guessing there is some way that package has 
| insisted that the install tries /usr/lib/R/library.

Agreed on the first bit.  I think /usr/lib/R/library -- where we as users
should never tread as it is /usr which belongs to apt/dpkg -- is just the
fallback.  Somehow the value got nuked.
 
| I don't know enough about the way that R and linux work agree the use of 
| those library location options. From ?.libPaths and a bit more hunting 
| around the R help I had the impression that the .libPath options would 
| be used in order. I think I can remove the /usr ones by hacking in 

Correct.

| /etc/R but that presumably risks losing access to the main libraries 
| that will have been installed there so that doesn't look a good step 
| forward except perhaps to debug things.

An easier option ... is to _explicitly add_ the
/usr/local/lib/R/site-library/ destination as an argument to install.packages!
 
| I suspect that I could do a complete remove of all of R, start again 
| installing r-cran-base, check that R installed by that updates fine run 
| by a user and then I could use synaptic to add the additional deb 
| packages you provide but doing that one by one and running R and 
| update.packages() after each one to see if I don't hit this until afetr 
| a particular deb but that's going to take quite a long time so I 
| wondered if you or the list had quicker suggestions to debug this.

That sounds too involved.
 
| Sorry for length but hope that's clearer (got to try to be an acceptable 
| friend on R lists as well as with linux and I really, really do 
| appreciate the generosity of this list).

No sweat. We're all just building karma :)

Dirk
Chris Evans
# 
Huge thanks to Johannes (sorry, Johannes, I think I implied that Dirk 
was doing all the hard work in rolling things in my earlier reply, the 
"you" is plural!)

What I've done:
A) removed r-cran-rgl and r-cran-xml using synaptic as root
B) re-entered new R session as user chris, confirmed that the packages 
had gone using installed.packages() and grep()!
C) ran "adduser chris staff" as root
D) logged off gnome session (ah, that bit's a pain when making group 
membership changes which won't otherwise propagate to current session!)
E) logged back in!
F) checked I was now member of staff: yes!
G) checked I can write to /usr/local/lib/R/site-library: yes!
H) ran R as user chris ... now it gets interesting and frustrating:
     no damn joy, update.packages and selecting only abind to update 
without specifying instlib says it runs fine but next time I run 
update.packages it still shows the old version and wants to do the 
update again so I
I) (and this was a debugging mistake for which I apologise) did the same 
but specifying instlib as /usr/local ... and it runs fine again (this 
time I checked that it had updated in /usr/local/... and there it was in 
solitary splendour as my first updated library/package there) ... but  ...
J) that hasn't gained me anything (I suspect the previous update without 
the "instlib=" may have been fine too) as update.packages still wants to 
update it ...
K) so I get thoughtful: clearly update.packages is reading from /usr/lib 
not /usr/local ...
L) so I check .libPaths():
 > .libPaths()
[1] "/home/chris/R/i486-pc-linux-gnu-library/2.13"
[2] "/usr/local/lib/R/site-library"
[3] "/usr/lib/R/site-library"
[4] "/usr/lib/R/library"
M) Hm.  Looks good to me but I see Dirk is telling me /usr/local/... 
should be first so I go to /etc/R/Renviron and I find:
R_LIBS_SITE=${R_LIBS_SITE-'/usr/local/lib/R/site-library:/usr/lib/R/site-library:/usr/lib/R/library'}
and nothing I can see in Renviron.site that would override that.

I _THINK_ I see a logic:  installation is happening "first come, first 
used" on the libPaths but update.packages() is reading from all the 
library locations and the later are overriting the earlier ... that 
would explain things but surely it would mean that no-one, including 
Dirk, would be able to run update.packages() successfully as a non-root 
user.

So now I'm baffled as to how to be friendly with R and linux/Debian on 
this machine!

Perhaps I should say that I would like to work by:
1) having the cran deb line in /etc/apt/sources.list and doing basic 
installation of R from there as my understanding is that that will get 
me the latest R for Debian when you (plural) have rolled it and put it 
on CRAN getting me round the downloading and replacing I used to do on 
windoze and also making it pretty likely that you'll have sorted out 
dependency problems etc.
2) would like then to rely on update.packages() from within R
3) but running as local user as I can see the logic of that

#1 and #2 are what I've always done on my Debian web server (where I was 
running some cgi-bin scripts invoking R) but I always did the 
update.packages() as root.  I've now moved that server to a shell 
account at mythic-beasts (great guys) so I'll have to do #3 for that too 
in future and though I could update as root on this laptop I do see the 
logic of not doing that and clearly you both and presumably many others 
on this list do #1, #2 and #3 so why is it not working for me?!

Suggestions gratefully received!

Very best & karma to all,

Chris



Dirk Eddelbuettel sent the following  at 21/09/11 13:49:

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Johannes Ranke
    

    

      
      #
    

  


  

      
Zitat von Chris Evans <chrishold at psyctc.org>:

            

      
      
      
    

        
Never mind - you are absolutely right that Dirk is doing the heavy  
lifting, i.e. packaging and I just recompile some of his packages for  
Debian stable.

            

      
      
      
    

        
Where does abind live on your system? update.packages() always  
mentions where it finds its candidates.

            

      
      
      
    

        
So at some point in time you let update.packages() create a directory  
in your home directory that would have precedence over system wide  
installations.

            

      
      
      
    

        
Yes, you and I can run update.packages() successfully, but only on  
paths with write permissions. As it does not seem a good idea to give  
a normal user write permissions on /usr/lib/ anywhere, I would suggest  
doing (from within R)

update.packages(.libPaths()[1])
update.packages(.libPaths()[2])

This has just worked nicely for me on my Debian stable laptop.

The other packages should (please correct me if I am wrong Dirk)  
remain under control of the Debian package management.

If you want to update such packages as well, you need to bug me or  
help me in creating those backports (or use apt pinning e.g. to  
testing, but I think this would make things too complicated). For  
example, I only update R base packages on CRAN/Debian (like  
r-cran-foreign, r-cran-nlme etc.) when a new R version is released.  
And there are a lot of packages that Dirk meticulously updates in  
Debian unstable that I have never backported for Debian stable, like  
r-cran-xml and r-cran-rgl.

Regards,

Johannes

P.S.: I also mainly use my own R distribution for Debian stable on  
servers, just recently started to use it on my laptop (going back from  
Ubuntu).

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Chris Evans
    

    

      
      #
    

  


  

      
Oh dear.  The saga continues.  See below.

jranke at uni-bremen.de sent the following  at 22/09/11 12:52:

            

      
      
      
    

        
Yes.  I think the first time I ran the update.packages() it asked me if 
I'd like to create that directory and I said yes.

            

      
      
      
    

        
I think on my machine the equivalent is update.packages(.libPaths()[2]) 
and then [3].  I did run [1] and [2] and the first run prompted me for 
the CRAN repository and then exited silently and instantly: I assume 
there's no package in my /home/... directory that needs updating.  Then 
I ran for .libPaths()[2] and it no longer needed to prompt for 
repository and exited silently and instantly.  Then I ran for 
.libPaths()[3] and it asked me about many libraries that could be 
updated starting with abind telling me it was in /usr/lib/R/site-library 
and at version 1.1.0 with 1.3.0 available via CRAN but when I asked it 
to upgrade abind it failed complaining that it couldn't write to the 
directory.  So what I did then was:

update.packages(.libPaths()[3],instlib="/usr/local/lib/R/site-lib")

That seemed to run fine, installed to /usr/local/... and I confess I've 
now gone into /usr/lib/R/... and deleted the now outdated libraries 
there and then used mv to move the remaining ones to /usr/local/...

All that begs the question how I got myself here.  I think the learning 
point is that there may be a way to make the instructions in the deb 
packages in stable (well, in the next stable I guess) spell out the 
relationship between the deb repository in stable, the deb repository in 
CRAN (with a link to the page in CRAN about the debs there) and the 
non-deb libraries in CRAN.  You might spell out options which I have as:
1) you can stay with the debs in stable, and update within R as an 
ordinary user, you won't get automatic R updates but you should get the 
package updates that are possible for that version of R
2) you can go with the debs in CRAN which will usually give you the 
latest version of R shortly after it is released.  In that  case [just 
my take and I can see that I could be quite wrong] you are best off 
doing  a complete remove of the version of R you installed from your 
Debian stable mirror using synaptic (or your own preferred package 
handler), _before_ adding the necessary line in /etc/apt/sources.list to 
get your local CRAN deb repository, after which you can do a clean 
install of R
3) you are probably wise to do adduser yourself staff to get write 
access to /usr/local/lib/R/site-library/
4) you should then find that things update fine through R and 
update.packages() running R as an ordinary user and you will be getting 
the latest R very soon after each upgrade and the latest upgrades of the 
packages you have installed.

It sounds to me as if there's another option which is use the debian 
backports deb repository/mirror but I'm not sure I understand the 
relationship between that option and the option of going to the CRAN debs.

Very best all & thanks again to you both Dirk & Johannes,

Chris


... rest of Johannes's reply and the thread follows for the record ...