Building R on Windows: mkdir of Rtools creates directories with read-only permissions [WEIRD]

Le lundi 30 juin 2014 ? 15:32 -0400, Duncan Murdoch a ?crit :
On 30/06/2014 2:41 PM, Milan Bouchet-Valat wrote:
Le lundi 30 juin 2014 ? 05:23 -0400, Duncan Murdoch a ?crit :
On 30/06/2014, 4:44 AM, Milan Bouchet-Valat wrote:
On Thu Jan 9 2014 03:47 Henrik Bengtsson wrote:
This is is an issue that bugged me for a while.  I encountered a year
ago (April 2012) when I first tried to build R from source on Windows.
 I never figured out what the solution is or if I'm doing something
wrong myself (but I have found a tedious workaround).  I'm still on
the same Windows 7 Ultimate machine with NTFS, but I now made sure I
started from scratch so I have a completely fresh setup (see details
at the end).

PROBLEM:
At the very first step I do:

C:\R\src\gnuwin32>make all
[... Waiting. No errors until ...]
cp R.dll ../../bin/i386
-------- Building ../../../bin/i386/Rblas.dll --------
gcc -std=gnu99  -shared  -o ../../../bin/i386/Rblas.dll blas.o
cmplxblas.o ../../gnuwin32/dllversion.o Rblas.def -L../../../bin/i386
-lR -lgfortran
c:/rtools/gcc-4.6.3/bin/../lib/gcc/i686-w64-mingw32/4.6.3/../../../../i686-w64-mingw32/bin/ld.exe:
cannot find -lR
collect2: ld returned 1 exit status
make[2]: *** [../../../bin/i386/Rblas.dll] Error 1
make[1]: *** [Rblas] Error 2
make: *** [rbuild] Error 2

However:

C:\R\src\gnuwin32>dir ..\..\bin\i386\
 Volume in drive C is Windows7_OS
 Volume Serial Number is E038-51CC

 Directory of C:\R\bin\i386

01/08/2014  06:18 PM    <DIR>          .
01/08/2014  06:18 PM    <DIR>          ..
01/08/2014  06:18 PM         3,059,712 R.dll
01/08/2014  06:18 PM           348,995 Rgraphapp.dll
01/08/2014  06:18 PM           102,975 Riconv.dll
01/08/2014  06:18 PM           154,917 Rzlib.dll
               4 File(s)      3,666,599 bytes
               2 Dir(s)  22,424,739,840 bytes free

What's weird is that these files have **read permission disabled**:

C:\R\src\gnuwin32>file ..\..\bin\i386\*.dll
..\..\bin\i386\R.dll:         writable, executable, regular file, no
read permission
..\..\bin\i386\Rgraphapp.dll: writable, executable, regular file, no
read permission
..\..\bin\i386\Riconv.dll:    writable, executable, regular file, no
read permission
..\..\bin\i386\Rzlib.dll:     writable, executable, regular file, no
read permission

What is going on?  Have anyone else seen this?  I've also tried
running as Administrator - no difference.
Hi Henrik,

Old thread, but I think I'm experiencing exactly the same problem. I
tried running as administrator and as a normal user, and no luck. I
noticed, though, that resetting permissions on the build tree makes the
error go away, and replaces it with another one stating that headers in
include/ could not be created due to permission issues.

This is on a machine where I was able to build R 3.0.1 successfully,
though I also remember having permission issues I eventually fixed by
resetting permissions on the build tree. Not sure what's changed since
then...

To add bit of information to the debugging you've already done: I
noticed that when opening the directory/file property dialog, a user
with a very long name including letters and numbers appears in the
permissions list for a second, and then disappears. This user reappears
every time I run 'make'. So maybe 'mkdir' or other tools use wrong user
IDs, creating bogus users. Just a guess.

Have you found a solution or workaround since January? I really need to
build an R installer and I seem to be stuck...
I have seen this, but I don't see it on every machine.  One one where I
do see it is the Windows builder machine in Dortmund.  The following is
inserted into the script after building the 32 bit build

cacls %name32% /T /E /G VORDEFINIERT\Benutzer:R > NUL

and similar after the 64 bit build.  (I don't remember whether
VORDEFINIERT\Benutzer is the current user name, or is some generic thing
saying "current user".  I think Uwe Ligges came up with this...)
Thanks. VORDEFINIERT\Benutzer appears to mean BUILTIN\Users (BUILTIN
\Utilisateurs in French), and refers to any user on the system (more or
less).

Unfortunately, I've run that, and it doesn't make any difference to the
error I'm seeing. But I don't understand what you mean by "after
building the 32 bit build" since the problem is precisely that this step
does not complete successfully.

Here is what I get:

C:\Windows\system32>cd C:\Rbuild\R-3.1.0-32bit

C:\Rbuild\R-3.1.0-32bit>cacls %name32% /T /E /G BUILTIN\UTILISATEURS:R > NUL

C:\Rbuild\R-3.1.0-32bit>cacls %name32% /T /E /G BUILTIN\UTILISATEURS:W > NUL

C:\Rbuild\R-3.1.0-32bit>cd src/gnuwin32

C:\Rbuild\R-3.1.0-32bit\src\gnuwin32>set TMPDIR=C:\Rbuild\tmp

C:\Rbuild\R-3.1.0-32bit\src\gnuwin32>make -j4 all
make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule
I doubt if this warning is the cause of your troubles, but I wouldn't 
bother with the -j4 option because of it.

On a more positive note:  our scripts are more complicated than yours. 
I've cc'd a copy of our R-devel script.  It has lots of hard-coded 
directories in it, so it would take some work to get it going, but maybe 
it'll be helpful.
Thanks, it's very interesting to see the full script you are using. It
may be worth adding a link to it in the docs, I would have used it
instead of my half-automated solution if I had seen it.

I've taken inspiration from it and run the build again, in a different
directory, and it succeeded. Though it seems the real difference is the
directory: for some obscure reason I suspect the old one contaminated
the whole tree below it with its broken permission, even if I had
removed the source tree itself.

So, if anybody in the future is in the same situation: start from an
entirely different directory under e.g. C:\.

Thanks for the support
Duncan Murdoch
.
make[1]: `MkRules' is up to date.
make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent make rule
.
make[4]: Nothing to be done for `svnonly'.
installing C headers
make[2]: Nothing to be done for `all'.
make[2]: `libRblas.dll.a' is up to date.
make[5]: Nothing to be done for `svnonly'.
installing C headers
make --no-print-directory -C ../extra/intl CFLAGS='-O3 -Wall -pedantic -mtune=core2' -f Makefile.win
make --no-print-directory -C ../appl CFLAGS='-O3 -Wall -pedantic -mtune=core2' FFLAGS='-O3 -mtune=core2' -f Makefile.win
make --no-print-directory -C ../nmath CFLAGS='-O3 -Wall -pedantic -mtune=core2'
FFLAGS='-O3 -mtune=core2' -f Makefile.win
make --no-print-directory -C ../main CFLAGS='-O3 -Wall -pedantic -mtune=core2' FFLAGS='-O3 -mtune=core2' malloc-DEFS='-DLEA_MALLOC' -f Makefile.win
gcc -std=gnu99 -m32 -I. -I../include -DHAVE_CONFIG_H -DR_DLL_BUILD -DR_ARCH='"i386"' -I../extra/xdr -O3 -Wall -pedantic -mtune=core2   -c serialize.c -o serialize.o
In file included from serialize.c:28:0:
../include/Defn.h:1244:21: fatal error: ../include/libintl.h: Permission denied
compilation terminated.
make[4]: *** [serialize.o] Error 1
make[3]: *** [rlibs] Error 2
make[2]: *** [../../bin/i386/R.dll] Error 2
make[1]: *** [rbuild] Error 2
make: *** [all] Error 2

C:\Rbuild\R-3.1.0-32bit\src\gnuwin32>ls ../include/libintl.h -l
---------- 1 Milan mkpasswd 14384 Jun 29 20:15 ../include/libintl.h

I see this weird permission pattern for most of the files in the build
tree. Running
cacls %name32% /T /E /G BUILTIN\UTILISATEURS:R > NUL
does not have any effect on the output of 'ls -l'.

Regards

Duncan Murdoch

Thanks for your help

TROUBLESHOOTING / WORKAROUND:
It appears that 'mkdir' (of Rtools) causes this problem, because if I
manually pre-create 'C:\R\src\gnuwin32\bin\i386' using Windows' mkdir
it works, e.g.

C:\R\src\gnuwin32>rm -fR ..\..\bin\
C:\R\src\gnuwin32>cmd /C mkdir ..\..\bin\i386
C:\R\src\gnuwin32>make rbuild
[...]
-------- Building ../../../bin/i386/Rblas.dll --------
gcc -std=gnu99  -shared  -o ../../../bin/i386/Rblas.dll blas.o
cmplxblas.o ../../gnuwin32/dllversion.o Rblas.def -L../../../bin/i386
-lR -lgfortran
make --no-print-directory -C front-ends
mkdir -p ../../../bin/i386
cp Rgui.exe ../../../bin/i386/Rgui.exe
mkdir -p ../../../bin/i386
cp Rterm.exe ../../../bin/i386/Rterm.exe
mkdir -p ../../../bin/i386
cp Rcmd.exe ../../../bin/i386/Rcmd.exe
mkdir -p ../../../bin/i386
cp RSetReg.exe ../../../bin/i386/RSetReg.exe
mkdir -p ../../../bin/i386
cp R.exe ../../../bin/i386/R.exe
mkdir -p ../../../bin/i386
cp Rscript.exe ../../../bin/i386/Rscript.exe
mkdir -p ../../../bin/i386
cp open.exe ../../../bin/i386/open.exe
mkdir -p ../../../bin
cp Rfe.exe ../../../bin/R.exe
mkdir -p ../../../bin
cp Rfe.exe ../../../bin/Rscript.exe
make[1]: `COPYRIGHTS' is up to date.
make --no-print-directory -C ../modules -f Makefile.win \
  CFLAGS='-O3 -Wall -pedantic -mtune=core2' FFLAGS='-O3 -mtune=core2'
gcc -std=gnu99  -shared -s -o ../../../bin/i386/Rlapack.dll dlamch.o
dlapack.o cmplx.o init_win.o Rlapackrc.o -L../../../bin/i386 -lR
-lRblas -lgfortran
cp lapack.dll ../../../modules/i386/lapack.dll

and there is (obviously) read permissions on those files:

C:\R\src\gnuwin32>file ..\..\bin\i386\*.dll
..\..\bin\i386\R.dll:         PE32 executable for MS Windows (DLL)
(GUI) Intel 80386 32-bit
..\..\bin\i386\Rblas.dll:     PE32 executable for MS Windows (DLL)
(console) Intel 80386 32-bit
..\..\bin\i386\Rgraphapp.dll: PE32 executable for MS Windows (DLL)
(GUI) Intel 80386 32-bit
..\..\bin\i386\Riconv.dll:    PE32 executable for MS Windows (DLL)
(console) Intel 80386 32-bit
..\..\bin\i386\Rlapack.dll:   PE32 executable for MS Windows (DLL)
(console) Intel 80386 32-bit
..\..\bin\i386\Rzlib.dll:     PE32 executable for MS Windows (DLL)
(console) Intel 80386 32-bit

Pre-creating the proper directories is tedious and only works up to
the point of 'make distribution', where I haven't found a similar
workaround (without modifying the Makefiles).  Note, I can get to the
point where I have build and check everything (including recommended
packages).

Just to convince you that mkdir is the problem, I can reproduce the
problem from this point, by removing the bin/ directory again and
relying on Rtools mkdir again;

C:\R\src\gnuwin32>rm -fR ..\..\bin
C:\R\src\gnuwin32>make rbuild
make[3]: Nothing to be done for `svnonly'.
installing C headers
make[1]: `libRblas.dll.a' is up to date.
make[4]: Nothing to be done for `svnonly'.
installing C headers
make --no-print-directory -C ../extra/intl CFLAGS='-O3 -Wall -pedantic
-mtune=core2' -f Makefile.win
make --no-print-directory -C ../appl CFLAGS='-O3 -Wall -pedantic
-mtune=core2' FFLAGS='-O3 -mtune=core2' -f Makefile.win
make --no-print-directory -C ../nmath CFLAGS='-O3 -Wall -pedantic
-mtune=core2' FFLAGS='-O3 -mtune=core2' -f Makefile.win
make --no-print-directory -C ../main CFLAGS='-O3 -Wall -pedantic
-mtune=core2' FFLAGS='-O3 -mtune=core2' malloc-DEFS='-DLEA_MALLOC' -f
Makefile.win
make --no-print-directory -C ./getline CFLAGS='-O3 -Wall -pedantic -mtune=core2'

make[3]: `gl.a' is up to date.
make -f Makefile.win makeMakedeps
make -f Makefile.win libpcre.a
make[4]: `libpcre.a' is up to date.
make[3]: Nothing to be done for `all'.
make -f Makefile.win makeMakedeps
make -f Makefile.win libtre.a
make[4]: `libtre.a' is up to date.
make[3]: Nothing to be done for `all'.
make[4]: `stamp' is up to date.
make[4]: `liblzma.a' is up to date.
make[2]: `R.dll' is up to date.
cp R.dll ../../bin/i386
-------- Building ../../../bin/i386/Rblas.dll --------
gcc -std=gnu99  -shared  -o ../../../bin/i386/Rblas.dll blas.o
cmplxblas.o ../../gnuwin32/dllversion.o Rblas.def -L../../../bin/i386
-lR -lgfortran
c:/rtools/gcc-4.6.3/bin/../lib/gcc/i686-w64-mingw32/4.6.3/../../../../i686-w64-mingw32/bin/ld.exe:
cannot find -lR
collect2: ld returned 1 exit status
make[2]: *** [../../../bin/i386/Rblas.dll] Error 1
make[1]: *** [Rblas] Error 2
make: *** [rbuild] Error 2

MORE TROUBLESHOOTING:
The MD5 checksum when it fails (reproducible):
C:\R\src\gnuwin32>md5sum ../../bin/i386/*.dll
75b2e64d3847e768631681922df5a81d *../../bin/i386/R.dll
12a35b63a28ad0e0fee8cae04f76e907 *../../bin/i386/Rgraphapp.dll
e2c40ae15d13dc17af33c1367f78817d *../../bin/i386/Riconv.dll
c5ea5a5cd2b7d3c07970f5314ce7296a *../../bin/i386/Rzlib.dll

The MD5 checksum when it works (reproducible):
C:\R\src\gnuwin32>md5sum ../../bin/i386/*.dll
75b2e64d3847e768631681922df5a81d *../../bin/i386/R.dll
12a35b63a28ad0e0fee8cae04f76e907 *../../bin/i386/Rgraphapp.dll
e2c40ae15d13dc17af33c1367f78817d *../../bin/i386/Riconv.dll
c5ea5a5cd2b7d3c07970f5314ce7296a *../../bin/i386/Rzlib.dll
+
c81b3399027ec9c24ca80ba92199327c *../../bin/i386/Rblas.dll
20d32cee2e561d1826af1a538844472b *../../bin/i386/Rlapack.dll

Thanks for any help,

Henrik

SETUP:
1. Downloaded/svn checkout the R source to C:\R\.
2. Installed most recent Rtools to C:\Rtools (completely removed previously)
(3. Modifying the MkRules.* files has no impact for reproducing this
problem.  Same with 'make rsync-recommended')

C:\R\src\gnuwin32>path
PATH=;C:\Rtools\bin;C:\Rtools\gcc-4.6.3\bin;C:\Windows;C:\Windows\system32

C:\R\src\gnuwin32>cat C:\Rtools\VERSION.txt
Rtools version 3.1.0.1939

C:\R\src\gnuwin32>ls ../..
COPYING      Makefile.fw  VERSION       configure.ac  po     tmp
ChangeLog    Makefile.in  VERSION-NICK  doc           share  tools
INSTALL      README       config.site   etc           src
Makeconf.in  Tcl          configure     m4            tests

______________________________________________
R-devel at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel