Skip to content

Assistance much appreciated

16 messages · Michael Felt, Duncan Murdoch, Peter Dalgaard +1 more

Original
Michael Felt
# 
I have been struggling with this error message - and think I finally 
understand it's context.

Start

Line by line debugging shows me the function works:

...
 > saveRDS(val, mapfile)
 > val
$variables
$variables$IANA_HTTP_status_code_db
[1]    0 1256

$variables$IANA_URI_scheme_db
[1] 1256 3458

$variables$table_of_HTTP_status_codes
[1] 4714  830


$references
named list()

$compressed
[1] TRUE

 > mapfile
[1] "./library/tools/R/sysdata.rdx"
 > [1] + Stopped (SIGTSTP)         R_DEFAULT_PACKAGES=NULL LC_ALL=C 
./bin/R --vanilla
root at x065:[/data/prj/cran/64/R-3.2.3]ls -l ./library/tools/R/sysdata.rdx
-rw-r--r--   1 root     33              193 Dec 17 16:49 
./library/tools/R/sysdata.rdx
root at x065:[/data/prj/cran/64/R-3.2.3]cat ./library/tools/R/sysdata.rdx
?b```b`fbf" SH0#h^?<??@?Al?&_(
                               ??(R?????        ?Y?sS
C7?PG3@??j/D                                        ?aA 
OG??x??????????$??X>4?3?89#5IF?$1)'5>?
?00F??,?(dA1T?(5-?(5/![PZ\2?$?<root at x065:[/data/prj/cran/64/R-3.2.3]
root at x065:[/data/prj/cran/64/R-3.2.3]
root at x065:[/data/prj/cran/64/R-3.2.3]gzip -dc ./library/tools/R/sysdata.rdx
X
j>      names   IANA_HTTP_status_code_db IANA_URI_scheme_db      
table_of_HTTP_status_codes???
?               variables
references
compressed?root at x065:[/data/prj/cran/64/R-3.2.3]fg
  R_DEFAULT_PACKAGES=NULL LC_ALL=C ./bin/R --vanilla

 >
 > e
<environment: 110dbf620>
 > destDir
[1] "./library/tools/R"
 > compress
[1] TRUE
 > debug(tools:::makeLazyLoadDB(e, file.path(destDir, "sysdata"), 
compress = compress))
Error: Line starting 'Package: tools ...' is malformed!
 > tools:::makeLazyLoadDB(e, file.path(destDir, "sysdata"), compress = 
compress)
Error: Line starting 'Package: tools ...' is malformed!
 >

So, at the end - my gut-feeling is that the error messages is not about 
the arguments, but us related to the "tools:::foo".

Where should I look next? It goes a bit beyond me to know where in R 
"tools:::foo" is being parsed.
Guessing here - but might be related to the switch from 32-bit to 64-bit 
AND that PowerPC is BigEndian rather than LittleEndian? Could also be 
absolutely wrong, way off base, etc. --> shoot me down!

Thanks,
Michael
Michael Felt
# 
More experimenting with calling commands:
 > tools:::foobar()
Error: Line starting 'Package: tools ...' is malformed!
 > tools::foobar()
Error: Line starting 'Package: tools ...' is malformed!
 > Tools:::foobar()
Error in loadNamespace(name) : there is no package called 'Tools'
 > loadNamespace(tools)
Error in loadNamespace(tools) : object 'tools' not found
 > loadNamespace("tools")
Error: Line starting 'Package: tools ...' is malformed!
 >
On 17-Dec-15 18:02, Michael Felt wrote:
Duncan Murdoch
#
On 17/12/2015 9:06 AM, Michael Felt wrote:
These both do a loadNamespace("tools").
Those are expected, and unrelated.
This shouldn't happen.  In a previous post you listed a long list of 
warnings about packages not being loaded; that's likely a serious 
problem, but I don't know if it is the cause or a consequence of this 
one.  Something is seriously wrong with the startup.  You could try 
debugging setup_Rmainloop in <src/main/main.c>.  to see if any warnings 
are being generated there.  There shouldn't be any.

Duncan Murdoch
Peter Dalgaard
# 
Presumably the file in question is one of

Peter-Dalgaards-MacBook-Air:BUILD pd$ grep -r "^Package: tools" *
library/tools/DESCRIPTION:Package: tools
src/library/tools/DESCRIPTION:Package: tools

so the first thing I'd do is to have a good look at those files and see if they got somehow corrupted.

If they look sane, presumably the bit of code that checks it isn't... So that would be my next thing to check.

-pd

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Michael Felt
    

    

      
      #
    

  


  

      
On 2015-12-17 18:56, Duncan Murdoch wrote:

            

      
      
      
    

        
Actually, it is what helped me realize that the problem was with the 
parsing of the call, rather than the arguments.

            

      
      
      
    

        
This is during make - so there is very little - and the messages was 
because I was forgetting to set LC_ALL=CC before starting ./bin/R

The file I am looking at is src/main/dcf.c (where the message is 
generated) - your file suggestion may be better - skipping over a few 
bread crumbs so to speak.

More in my reply to Peter's comments.

            

      
      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Michael Felt
    

    

      
      #
    

  


  

      
On 2015-12-17 19:30, peter dalgaard wrote:

            

      
      
      
    

        
I had just gotten here, and tried comparing them with the 32-bit build 
(as I do not know what sane is yet).

Sources are in:
/data/prj/cran/R-3.2.3

Builds are in:
/data/prj/cran/32/R-3.2.3 and
/data/prj/cran/64/R-3.2.3

The files, in the build directories look "sane", at least comparable

root at x065:[/data/prj/cran/R-3.2.3]sdiff -w 132 
/data/prj/cran/??/R-3.2.3/library/tools/DESCRIPTION
Package: tools                                                      
Package: tools
Version: 3.2.3                                                      
Version: 3.2.3
Priority: base                                                      
Priority: base
Title: Tools for Package Development                                
Title: Tools for Package Development
Author: R Core Team                                                 
Author: R Core Team
Maintainer: R Core Team <R-core at r-project.org>                      
Maintainer: R Core Team <R-core at r-project.org>
Description: Tools for package development, administration and      
Description: Tools for package development, administration and
License: Part of R 3.2.3                                            
License: Part of R 3.2.3
Suggests: XML, codetools, methods                                   
Suggests: XML, codetools, methods
Built: R 3.2.3; powerpc-ibm-aix5.3.7.0; 2015-12-17 12:02:34 UTC  |  
Built: R 3.2.3; ; Thu Dec 17 17:39:55 UTC 2015; unix

As the 32-bit build completes, while the 64-bit stops at "tools" there 
is a difference in what they have available:

root at x065:[/data/prj/cran/R-3.2.3]ls -l 
/data/prj/cran/??/R-3.2.3/library/tools
/data/prj/cran/32/R-3.2.3/library/tools:
total 72
-rw-r--r--   1 root     33              356 Dec 17 12:02 DESCRIPTION
-rw-r--r--   1 root     33             3665 Dec 16 11:37 INDEX
drwxr-sr-x   2 root     33             4096 Dec 17 08:45 Meta
-rw-r--r--   1 root     33             7112 Dec 17 12:02 NAMESPACE
drwxr-sr-x   2 root     33             4096 Dec 17 12:02 R
drwxr-sr-x   2 root     33             4096 Dec 16 11:37 help
drwxr-sr-x   2 root     33             4096 Dec 16 11:37 html
drwxr-sr-x   2 root     33             4096 Dec 16 11:34 libs

/data/prj/cran/64/R-3.2.3/library/tools:
total 40
-rw-r--r--   1 root     33              339 Dec 17 17:40 DESCRIPTION
-rw-r--r--   1 root     33             7112 Dec 17 17:40 NAMESPACE
drwxr-sr-x   2 root     33             4096 Dec 17 17:19 R
drwxr-sr-x   2 root     33             4096 Dec 17 05:58 libs

So, is dcf.c, or main.c the better place to check. dcf.c does not look 
like "fun" - all those lexical arguments at the start:

SEXP attribute_hidden do_readDCF(SEXP call, SEXP op, SEXP args, SEXP env)
{
.... man many lines ....
                                } else {
                                        /* Must be a regular line with 
no tag ... */
                                        line[20] = '\0';
                                        error(_("Line starting '%s ...' 
is malformed!"), line);
                                }
                        }
                }
        }
... more lines to go...

Thanks for the assistance!

Michael

  
  


  


      

    

    

      
      

        


  

        

      Peter Dalgaard
    

    

      
      #
    

  


  

          

      
      
      
    

        
OK. Perhaps a check for special characters with a hex editor, just to be sure? But probably not the cause.

            

      
      
      
    

        
Cue story of looking for keys under lamppost...

            

      
      
      
    

        
As you're dying in an else clause, a previous if () must contain the clue. Unfortunately not necessarily the matching one.

My guess is that your TRE library is broken. The line should have matched the RE "regline" defined as

   tre_regcomp(&regline, "^[^:]+:[[:blank:]]*", REG_EXTENDED);
...and used here:
                if(tre_regexecb(&regline, line, 1, regmatch, 0) == 0) { 

but apparently does not.

            

      
      
      
    

        

  
    

      
      
    

  


  


      

    

    

      
      

        


  

        

      Simon Urbanek
    

    

      
      #
    

  


  

      
Michael,

I got access to PDP AIX so I can try to replicate your problem. Can you, please, share exactly your setup - AIX version and well as how exactly you installed the compilers (=where from)? I can then try to replicate it. AFAICS there is no official binary for gfortran nor gcc 4.7 so it must be some 3rd party - which could also be a problem.

Thanks,
Simon

        
On Dec 17, 2015, at 12:02 PM, Michael Felt <aixtools at gmail.com> wrote:

        

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Simon Urbanek
    

    

      
      #
    

  


  

      
FWIW I was able to replicate the problem using AIX7.1 on Power8, R 3.2.3, vac/xlc 11.1, xlf 13.1 in 64-bit mode. Indeed, Peter's analysis is correct re_regexecb(&regline, line..) returns REG_NOMATCH even though the string is literally "Package: tools" - no special characters anywhere.

        
On Dec 17, 2015, at 3:37 PM, peter dalgaard <pdalgd at gmail.com> wrote:

        

            

      
      
      
    

        

      
      
    

  
  


  


      

    

    

      
      

        


  

        

      Michael Felt
    

    

      
      #
    

  


  

      
On 2015-12-18 04:30, Simon Urbanek wrote:

            

      
      
      
    

        
Worth a great deal to me! Thank you both. Also glad it has been 
independently replicated. And using the IBM compilers. I was concerned 
it was a gcc/gfortran behavior difference.
As you are testing with the IBM compilers - are you also seeing the 
duplicate symbols from *yyl* (coming from in gramRd.y and gramLatex.y in 
tools). I have used both the gramRd.c and gramLatex.c provided, as well 
as new ones made with bison-3.0.8 and get the same problem on AIX 5.3 
(not gotten back to test on AIX 6.1 and AIX 7.1 - but the 32-bit 
version, built on AIX 5.3 runs fine on AIX 7.1).
I see a different behavior re: the definition of the "symbols" on AIX 
versus debian (wheezy) where debian just leaves it as an unitilised 
variable (.bss) but does not pre-define/establish it's global address.

p.s. are you also in a position to try on 64-bit Linux - and compare LE 
with BE (like AIX)?
And maybe it is a bug in bison/yacc that has never been properly explained.

So - we need to dissect re_regexecb() - looking in particular at 
differences that 32 versus 64-bit may have?!

Michael

  
  


  


      

    

    

      
      

        


  

        

      Michael Felt
    

    

      
      #
    

  


  

      
On 2015-12-17 21:37, peter dalgaard wrote:

            

      
      
      
    

        
int
tre_regcomp(regex_t *preg, const char *regex, int cflags)
{
   return tre_regncomp(preg, regex, regex ? strlen(regex) : 0, cflags);
}

The code is a bit too 'string busy' for me to be comfortable - so I took 
a different approach to look for potential differences in how 32-bit 
versus 64-bit were viewing things. Again, it may not be that the root 
cause is here - but I hope looking at the diff -u of the .i files gives 
someone an good impression of how the data types are being swapped about.

For one example - I had a very old program that had a typedef for 
"typedef unsigned long long ulonglong_t;" that would not parse well. 
When I changed that to unit64_t (as defined in /usr/include/sys/stdint.h)

#ifndef _STD_UINT64_T
#ifdef  __64BIT__
typedef unsigned long           uint64_t;
#else   /* _ILP32 */
#if defined(_LONG_LONG)
typedef unsigned long long      uint64_t;
#endif
#endif
#endif /* _STD_UINT64_T */


All my compilers and os mix (gcc, xlc, AIX and linux) were happy!

So, I am hoping something triggers a thought in the following: (or the 
haystack gets smaller...)
root at x072:[/]diff -u /data/prj/cran/??/R-3.2.3/src/extra/tre/regcomp.i
--- /data/prj/cran/32/R-3.2.3/src/extra/tre/regcomp.i   2015-12-18 
08:10:25.000000000 +0000
+++ /data/prj/cran/64/R-3.2.3/src/extra/tre/regcomp.i   2015-12-18 
08:10:58.000000000 +0000
@@ -1,5 +1,5 @@
   # 1 "/data/prj/cran/R-3.2.3/src/extra/tre/regcomp.c"
- # 1 "/data/prj/cran/32/R-3.2.3/src/extra/tre//"
+ # 1 "/data/prj/cran/64/R-3.2.3/src/extra/tre//"
   # 1 "<command-line>"
   # 1 "/data/prj/cran/R-3.2.3/src/extra/tre/regcomp.c"
   # 10 "/data/prj/cran/R-3.2.3/src/extra/tre/regcomp.c"
@@ -171,18 +171,11 @@
   typedef long int ptrdiff_t;
   # 119 
"/opt/lib/gcc/powerpc-ibm-aix5.3.0.0/4.7.4/include-fixed/sys/types.h" 3 4
   typedef unsigned int wctype_t;
+ # 128 
"/opt/lib/gcc/powerpc-ibm-aix5.3.0.0/4.7.4/include-fixed/sys/types.h" 3 4
+ typedef long fpos_t;



-
-
-
- typedef long long fpos_t;
-
-
-
-
-
   typedef long long fpos64_t;
   # 145 
"/opt/lib/gcc/powerpc-ibm-aix5.3.0.0/4.7.4/include-fixed/sys/types.h" 3 4
   typedef int time_t;
@@ -258,9 +251,9 @@



- typedef long long off_t;


+ typedef long off_t;



@@ -2785,8 +2778,8 @@
   extern char *tmpnam(char *);
   extern int fclose(FILE *);
   extern int fflush(FILE *);
- extern FILE * fopen64(const char *restrict, const char *restrict);
- extern FILE * freopen64(const char *restrict, const char *restrict, 
FILE *restrict);
+ extern FILE * fopen(const char *restrict, const char *restrict);
+ extern FILE * freopen(const char *restrict, const char *restrict, FILE 
*restrict);
   extern void setbuf(FILE *restrict, char *restrict);
   extern int setvbuf(FILE *restrict, char *restrict, int, size_t);
   extern int fprintf(FILE *restrict, const char *restrict, ...);
@@ -2816,9 +2809,9 @@
   extern char *gets(char *);
   extern int puts(const char *);
   extern int ungetc(int, FILE *);
- extern int fgetpos64(FILE *restrict, fpos_t *restrict);
+ extern int fgetpos(FILE *restrict, fpos_t *restrict);
   extern int fseek(FILE *, long int, int);
- extern int fsetpos64(FILE *, const fpos_t *);
+ extern int fsetpos(FILE *, const fpos_t *);
   extern long ftell(FILE *);
   extern void rewind(FILE *);
   extern void perror(const char *);
@@ -2841,8 +2834,8 @@

   extern void flockfile(FILE *);
   extern void funlockfile(FILE *);
- extern int fseeko64(FILE *, off_t, int);
- extern off_t ftello64(FILE *);
+ extern int fseeko(FILE *, off_t, int);
+ extern off_t ftello(FILE *);
   extern int ftrylockfile(FILE *);
   extern void funlockfile(FILE *);
   # 470 
"/opt/lib/gcc/powerpc-ibm-aix5.3.0.0/4.7.4/include-fixed/stdio.h" 3 4

  
  


  


      

    

    

      
      

        


  

        

      Michael Felt
    

    

      
      #
    

  


  

      
On 2015-12-18 02:29, Simon Urbanek wrote:

            

      
      
      
    

        
I built gcc and gfortran myself. Available via 
http://download.aixtools.net/test/aixtools.obj.gcc.4.7.4.0.aix537.I 
(still in the test directory because the fileset name should be 
aixtools.gnu.gcc) - other filesets and instructions are available at: 
http://download.aixtools.net/gnu/gcc/

Note, especially, I was not smart enough - then - to automatically run 
the mkheaders command (full command is either

  /opt/libexec/gcc/powerpc-ibm-aix5.3.7.0/4.7.4/install-tools/mkheaders or  /opt/libexec/gcc/powerpc-ibm-aix5.3.0.0/4.7.4/install-tools/mkheaders
depending on the version of libtool I had installed when packaging (the latest also looks at the TL level, hence 5.3.7.0 rather that 5.3.0.0)


Only looking at the rte (runtime) filesets - this is what I have added: 
All are available via http:://www.aixtools.net (wiki interface) or 
http://download.aixtools.net (self-service).

The AIXTOOLBOX approach uses rpm for packaging - which I did not 
understand when I got started several years back. And I am not switching 
to rpm - following in the footsteps of Perzl because:
a) Michael Perzl does an excellent job of keeping the AIXTOOLBOX where 
it should be - no need to replicate that
b) I hate the RPM dependency nightmare (e.g., installing three rpm -dev 
packages  just to get 5 include files)
c) The RPM.spec files often (well occaisionally, but once is once too 
many for me) overwrite AIX/IBM installp files - so updates of one damage 
the other
d) RPM and NIM did not work as well as NIM and INSTALLP

Almost end of soapbox --- In AIX 5.2 IBM included a tool called 
mkinstallp. I have a tool around that - that I call buildaix - that 
largely automates packaging OSS/FOSS, especially when the packaging 
follows GNU build guidelines.

root at x065:[/data/prj/cran/R-3.2.3/src/extra/tre] aixtools | grep rte <
   aixtools.32.R.rte          3.1.3.0    C     F    1548 1130 1457
   aixtools.bison.rte         3.0.4.0    C     F    bison 1504 0125 1445
   aixtools.buildaix.rte      2.0.3.1    C     F    1540 1007 2004
   aixtools.bzip2.rte         1.0.6.0    C     F    1543 1030 1351
   aixtools.curl.rte         7.45.0.0    C     F    1544 1106 1841
   aixtools.gmp.rte           5.0.5.0    C     F    gmp version 5.0.5 
for powerpc
   aixtools.gnu.bash.rte     4.3.42.0    C     F    1537 0913 1505
   aixtools.gnu.gettext.rte  0.19.5.0    C     F    1531 0807 0000
   aixtools.gnu.gzip.rte      1.6.0.0    C     F    1550 1216 1405
   aixtools.gnu.gzip.rte.64   1.6.0.0    C     F    1550 1216 1417
   aixtools.gnu.m4.rte       1.4.17.0    C     F    1523 0613 1221
   aixtools.gnu.make.rte      4.1.0.0    C     F    1523 0613 1158
   aixtools.libiconv.rte     1.14.0.0    C     F    libiconv 1447 1127 1728
   aixtools.libjpeg.rte       6.2.0.0    C     F    libjpeg version 
6.2.0 for
   aixtools.libpng.rte       1.5.12.0    C     F    libpng version 
1.5.12 for
   aixtools.mpc.rte           1.0.2.0    C     F    mpc 2014.Oct.31 
15.44 CUT
   aixtools.mpfr.rte          2.4.2.0    C     F    mpfr 2014.Oct.31 
15.19 CUT
   aixtools.obj.gcc.rte       4.7.4.0    C     F    1515 0416 2134
   aixtools.pcre.rte         8.37.0.0    C     F    1544 1106 1445
   aixtools.pkgconfig.gcc.rte
   aixtools.tiff.rte          4.0.6.0    C     F    1542 1024 2144
   aixtools.xz.rte            5.0.8.0    C     F    1522 0602 0809
   aixtools.zlib.rte          1.2.8.1    C     F    zlib 1505 0203 1627

In closing, I made the mistake of thinking people know installp as well 
as they know rpm (at command-line). If you do not know installp at the 
command line level
a) run rm -f /where/you/download/to/.toc after everytime you add a file
b) use "smit install" to install filesets

If you are comfortable with installp cli interface the standard commands 
to remember are:
a) inutoc /where/you/download/to
b) installp -d /where/you/download/to -L # to list filesets
c) installp -d /where/you/download/to -agX <what you want to install> | 
"all"

Part II

a) I download R-project files to /data/prj/cran/downloads
b) cd /data/prj/cran
     gzip -dc downloads/R-3.2.3.tar.gz | tar xf -
     cp -rp R-3.2.3 R-3.2.3.dist
c) ls -l 32/*.env 64/*.env

sdiff -w ??/aix.env
cat bin/my_shared.ksh

d) slibclean; rm -rf 32/R-3.2.3; rm -rf 64/R-3.2.3
     mkdir 32/R-3.2.3
     mkdir 64/R-3.2.3

e) note current diffs between R-3.2.3 and R-3.2.3.dist
root at x072:[/data/prj/cran]diff -ru R-3.2.3.dist R-3.2.3
diff -ru R-3.2.3.dist/configure R-3.2.3/configure
--- R-3.2.3.dist/configure      2015-12-02 23:16:46.000000000 +0000
+++ R-3.2.3/configure   2015-12-16 10:52:15.000000000 +0000
@@ -26339,11 +26339,11 @@
       ##ADD: However, for example, symbol in libc of memcpy is 
__memmove,__memmove64.
       ##ADD: This black magic puts lc before lR and pockets this.
       if test "x${OBJECT_MODE}" = "x64"; then
-       main_ldflags="${wl}-brtl ${wl}-bexpall ${wl}-bpT:0x100000000 
${wl}-bpD:0x110000000 -lc"
+       main_ldflags="${wl}-brtl ${wl}-bpT:0x100000000 
${wl}-bpD:0x110000000 -lc"
       else
-       main_ldflags="${wl}-brtl ${wl}-bexpall -lc"
+       main_ldflags="${wl}-brtl ${wl}-bmaxdata:0xd0000000/dsa -lc"
       fi
-     shlib_ldflags="${wl}-brtl ${wl}-G ${wl}-bexpall ${wl}-bnoentry -lc"
+     shlib_ldflags="${wl}-G ${wl}-bnoentry -lc"
       SHLIB_LIBADD="\$(LIBM)"
       shlib_cxxldflags="${shlib_ldflags}"
       if test "${GCC}" = yes; then
diff -ru R-3.2.3.dist/share/make/basepkg.mk R-3.2.3/share/make/basepkg.mk
--- R-3.2.3.dist/share/make/basepkg.mk  2015-03-18 23:04:12.000000000 +0000
+++ R-3.2.3/share/make/basepkg.mk       2015-12-17 12:01:02.000000000 +0000
@@ -148,7 +148,9 @@

   sysdata: $(srcdir)/R/sysdata.rda
         @$(ECHO) "installing 'sysdata.rda'"
-       @$(ECHO) 
"tools:::sysdata2LazyLoadDB(\"$(srcdir)/R/sysdata.rda\",\"$(top_builddir)/library/$(pkg)/R\")" 
| \
+       $(ECHO) "warnings()" | \
+         R_DEFAULT_PACKAGES=NULL LC_ALL=C $(R_EXE)
+       $(ECHO) 
"tools:::sysdata2LazyLoadDB(\"$(srcdir)/R/sysdata.rda\",\"$(top_builddir)/library/$(pkg)/R\")" 
| \
           R_DEFAULT_PACKAGES=NULL LC_ALL=C $(R_EXE)


diff -ru R-3.2.3.dist/src/include/Defn.h R-3.2.3/src/include/Defn.h
--- R-3.2.3.dist/src/include/Defn.h     2015-08-25 22:15:24.000000000 +0000
+++ R-3.2.3/src/include/Defn.h  2015-12-16 10:53:08.000000000 +0000
@@ -1271,7 +1271,7 @@
   void invalidate_cached_recodings(void);  /* from sysutils.c */
   void resetICUcollator(void); /* from util.c */
   void dt_invalidate_locale(); /* from Rstrptime.h */
- int R_OutputCon; /* from connections.c */
+ extern int R_OutputCon; /* from connections.c */
   extern int R_InitReadItemDepth, R_ReadItemDepth; /* from serialize.c */
   void get_current_mem(size_t *,size_t *,size_t *); /* from memory.c */
   unsigned long get_duplicate_counter(void);  /* from duplicate.c */
diff -ru R-3.2.3.dist/src/library/tools/R/makeLazyLoad.R 
R-3.2.3/src/library/tools/R/makeLazyLoad.R
--- R-3.2.3.dist/src/library/tools/R/makeLazyLoad.R     2015-08-25 
22:16:54.000000000 +0000
+++ R-3.2.3/src/library/tools/R/makeLazyLoad.R  2015-12-17 
06:53:34.000000000 +0000
@@ -38,9 +38,13 @@

   sysdata2LazyLoadDB <- function(srcFile, destDir, compress = TRUE)
   {
+ browser()
       e <- new.env(hash=TRUE)
+ # browser()
       load(srcFile, e)
+ # browser()
       makeLazyLoadDB(e, file.path(destDir, "sysdata"), compress = compress)
+ # browser()
   }

   list_data_in_pkg <- function(package, lib.loc = NULL, dataDir = NULL)
diff -ru R-3.2.3.dist/src/library/tools/src/gramLatex.c 
R-3.2.3/src/library/tools/src/gramLatex.c
--- R-3.2.3.dist/src/library/tools/src/gramLatex.c      2015-12-09 
23:20:01.000000000 +0000
+++ R-3.2.3/src/library/tools/src/gramLatex.c   2015-12-17 
14:50:14.000000000 +0000
@@ -1358,7 +1358,7 @@


   /* The lookahead symbol.  */
- int yychar;
+ extern int yychar;


   #ifndef YY_IGNORE_MAYBE_UNINITIALIZED_BEGIN
@@ -1370,7 +1370,7 @@
   #endif

   /* The semantic value of the lookahead symbol.  */
- YYSTYPE yylval YY_INITIAL_VALUE(yyval_default);
+ extern YYSTYPE yylval YY_INITIAL_VALUE(yyval_default);

   /* Location data for the lookahead symbol.  */
   YYLTYPE yylloc
@@ -1381,7 +1381,7 @@


   /* Number of syntax errors so far.  */
- int yynerrs;
+ extern int yynerrs;


   /*----------.
diff -ru R-3.2.3.dist/src/library/tools/src/gramRd.c 
R-3.2.3/src/library/tools/src/gramRd.c
--- R-3.2.3.dist/src/library/tools/src/gramRd.c 2015-12-09 
23:20:01.000000000 +0000
+++ R-3.2.3/src/library/tools/src/gramRd.c      2015-12-17 
14:51:19.000000000 +0000
@@ -1972,7 +1972,7 @@


   /* The lookahead symbol.  */
- int yychar;
+ extern int yychar;


   #ifndef YY_IGNORE_MAYBE_UNINITIALIZED_BEGIN
@@ -1984,7 +1984,7 @@
   #endif

   /* The semantic value of the lookahead symbol.  */
- YYSTYPE yylval YY_INITIAL_VALUE(yyval_default);
+ extern YYSTYPE yylval YY_INITIAL_VALUE(yyval_default);

   /* Location data for the lookahead symbol.  */
   YYLTYPE yylloc
@@ -1995,7 +1995,7 @@


   /* Number of syntax errors so far.  */
- int yynerrs;
+ extern int yynerrs;


   /*----------.

+++++++++
The changes in /src/library/tools/R/makeLazyLoad.R are not relevant 
(initial learning...)

f) cd 32/R-3.2.3; ../../bin/my_shared.ksh aix R-3.2.3; export 
OBJECT_MODE=32; make -j4; cd -
g) cd 64/R-3.2.3; ../../bin/my_shared.ksh aix R-3.2.3; export 
OBJECT_MODE=64; make -j4 && cd -

Michael

p.s. please keep me posted on any difficulties - so I can add better 
instructions on the wiki.
p.p.s. - I am using R-3.2.3 to by-pass the removal of the src/extra packages

  
  


  


      

    

    

      
      

        


  

        

      Michael Felt
    

    

      
      #
    

  


  

      
On 2015-12-18 10:58, Michael Felt wrote:

            

      
      
      
    

        
Forgot this part:

2015-12-18 10:00                   32 and 64 bit aix 
environments                   Page 1


export OBJECT_MODE=32                        export OBJECT_MODE=64
export CC="gcc -maix32 -mcpu=power5"         export AR="ar -X64"
export CXX="g++ -maix32 -mcpu=power5"        export CC="gcc -maix64 
-mcpu=power5"
export F77="gfortran -maix32 -mcpu=power5"   export CXX="g++ -maix64 
-mcpu=power5"
export FC="gfortran -maix32 -mcpu=power5"    export F77="gfortran 
-maix64 -mcpu=power5"
export LDFLAGS="-L/opt/lib"                  export FC="gfortran -maix64 
-mcpu=power5"
                                              export LDFLAGS="-L/opt/lib"


2015-12-16 11:26                bin/my_shared.ksh                 Page 1


#!/usr/bin/ksh
# my_config_test
#       --with-cairo=no --with-libpng=no --with-jpeglib=no 
--with-libtiff=no \
#       --with-cairo=no --with-libpng=yes --with-jpeglib=yes 
--with-libtiff=yes \
#       --with-libpth-prefix=/opt \
#       --with-aix-soname=aix
HOST=$1
RELEASE=$2

. ../${HOST}.env || exit -1

/data/prj/cran/${RELEASE}/configure --disable-lto --prefix=/opt \
         --enable-R-shlib \
         --with-cairo=no --with-libpng=no --with-jpeglib=no 
--with-libtiff=no \
         --with-readline=no --with-x=no

  
  


  


      

    

    

      
      

        


  

        

      Michael Felt
    

    

      
      #
    

  


  

      
On 2015-12-18 02:29, Simon Urbanek wrote:

            

      
      
      
    

        
I installed a fresh system - AIX 6.1 TL5 this time, and built the system 
from scratch.

It is a long read, and I also made some discoveries about the contents 
of some of my newer packaging.
However, it you go through it you will see what is needed to make it all 
work.

The starting point are these filesets - available via my aixtools.net site:

root at x069:[/data/prj/cran/32/R-aix-3.2.3]ls -l /tmp/download
total 622400
-rw-r--r--    1 root     system        27482 Dec 18 12:34 .toc
-rwxrwxr--    1 199      33           828928 Sep 17 2012  
aixtools.gmp.5.0.5.0.I
-rw-r--r--    1 root     system      2102784 Jul 30 07:04 
aixtools.gnu.autoconf.2.69.0.0.aix537.I
-rw-r--r--    1 root     system      1461248 Jul 30 07:56 
aixtools.gnu.automake.1.15.0.0.aix537.I
-rw-r--r--    1 root     system      5321728 Sep 13 15:10 
aixtools.gnu.bash.4.3.42.0.aix537.I
-rw-r--r--    1 root     system     11640832 Aug 07 00:02 
aixtools.gnu.gettext.0.19.5.0.aix537.I
-rw-r--r--    1 root     system      1702912 Jul 30 08:08 
aixtools.gnu.libtool.2.4.6.0.aix537.I
-rw-r--r--    1 root     system       541184 Jun 13 2015  
aixtools.gnu.m4.1.4.17.0.aix537.I
-rw-r--r--    1 root     system      1229824 Jun 13 2015  
aixtools.gnu.make.4.1.0.0.aix537.I
-rw-r--r--    1 root     system      1348096 Nov 27 2014  
aixtools.libiconv.1.14.0.0.aix537.I
-rw-r--r--    1 199      33           265216 Oct 31 2014  
aixtools.mpc.1.0.2.0.I
-rw-r--r--    1 199      33           576512 Oct 31 2014  
aixtools.mpfr.2.4.2.0.I
-rw-r--r--    1 199      1954      290915328 Apr 16 2015  
aixtools.obj.gcc.4.7.4.0.aix537.I
-rw-r--r--    1 root     system       679424 Jun 02 2015  
aixtools.xz.5.0.8.0.aix537.I


Note: the aixtools.xz (lzma support) was a surprise, as the internal one 
did not link properly, for some reason (all in the file,
not included here, but available at: 
http://download.aixtools.net/test/R/build-freshstart.text (Trying to not 
overfill lots of mail boxes).

Also note, in the file there is a segmentation fault, core dump, but 
does not halt the build. Maybe this has been around on my previous 
builds on AIX 5.3 and 7.1 - but not noticed.

I did struggle for a few days with a buggy bos.adt.libm on AIX 7.1 
(installing an older version made my problem go away).

Segmentation error text:

  *** caught segfault ***
address 55544300, cause 'memory not mapped'

Traceback:
  1: format.POSIXlt(as.POSIXlt(x, tz), format, usetz, ...)
  2: structure(format.POSIXlt(as.POSIXlt(x, tz), format, usetz, 
...),     names = names(x))
  3: format.POSIXct(Sys.time(), "%Y-%m-%d at %H:%M:%S")
  4: format(Sys.time(), "%Y-%m-%d at %H:%M:%S")
  5: eval(expr, envir, enclos)
  6: eval(expr, env)
  7: withVisible(eval(expr, env))
  8: doTryCatch(return(expr), name, parentenv, handler)
  9: tryCatchOne(expr, names, parentenv, handlers[[1L]])
10: tryCatchList(expr, classes, parentenv, handlers)
11: tryCatch(withVisible(eval(expr, env)), error = function(e) e)
12: evalWithOpt(ce, options, env)
13: processRdChunk(block, stage, options, env, macros = macros)
14: recurse(block[[i]])
15: recurse(block[[i]])
16: recurse(block[[i]])
17: recurse(block)
18: getDynamicFlags(block)
19: recurse(block)
20: expandDynamicFlags(recurse(block), options)
21: processRdSexprs(Rd, stage, options, macros = attr(Rd, "macros"))
22: prepare_Rd(f, encoding = encoding, defines = os, stages = 
stages,     warningCalls = FALSE, stage2 = step > 1L, stage3 = step 
 >         2L, macros = macros)
23: FUN(X[[i]], ...)
24: lapply(files, .fetch_Rd_object)
25: .build_Rd_db(dir, manfiles, db_file = db_file, encoding = 
encoding,     built_file = built_file)
26: 
tools:::.install_package_Rd_objects("/data/prj/cran/R-3.2.3/src/library/tools",     
"../../library/tools")
aborting ...
/bin/sh[3]: 4915208 Segmentation fault(coredump)
Makefile:73: recipe for target 'tools.Rdts' failed
make[3]: *** [tools.Rdts] Error 1
make[3]: Leaving directory '/data/prj/cran/32/R-aix-3.2.3/src/library'
Makefile:79: recipe for target 'Rdobjects' failed
make[2]: *** [Rdobjects] Error 2
make[2]: Leaving directory '/data/prj/cran/32/R-aix-3.2.3/src/library'
Makefile:62: recipe for target 'docs' failed
make[1]: *** [docs] Error 2
make[1]: Leaving directory '/data/prj/cran/32/R-aix-3.2.3/src/library'
Makefile:72: recipe for target 'docs' failed
make: [docs] Error 2 (ignored)
make[1]: Entering directory 
'/data/prj/cran/32/R-aix-3.2.3/src/library/Recommended'
make[2]: Entering directory 
'/data/prj/cran/32/R-aix-3.2.3/src/library/Recommended'
begin installing recommended package MASS
* installing *source* package 'MASS' ...
** package 'MASS' successfully unpacked and MD5 sums checked
** libs
....

Simon - Thanks again for testing/checking,
Michael

  
  


  


      

    

    

      
      

        


  

        

      Michael Felt
    

    

      
      #
    

  


  

      
On 2015-12-18 14:21, Michael Felt wrote:

            

      
      
      
    

        
To simplify this, I put this is a compressed archive at: 
http://download.aixtools.net/test/R/download-aixtools.tar.bz2

  
  


  


      

    

    
3 days later

    

      
      

        


  

        

      Michael Felt
    

    

      
      #
    

  


  

      
On 2015-12-18 09:26, Michael Felt wrote:

            

      
      
      
    

        
I wonder if it could be
Looking here, I saw this line - and I am wondering if L'0' as 64-bit is 
too long, compared to 32-bit, especially since I read somnewhere that 
the strings are being processed byte by byte. (note: on Linux L'0' is 
the same size in both 32 and 64-bit objects).


+3562 wregex[wlen] = L'\0';
+3563 ret = tre_compile(preg, wregex, wlen, cflags);
+3564 free(wregex);
+3565
+3566
+3567
+3568
+3569 return ret;
+3570 }
+3571
"regcomp.i" The cursor is at line 3570 of 3644 --97%-- .


From:
+102 #endif /* TRE_MULTIBYTE */
+103
+104 wregex[wlen] = L'\0';
+105 ret = tre_compile(preg, wregex, wlen, cflags);
+106 xfree(wregex);
+107 #else /* !TRE_WCHAR */
+108 ret = tre_compile(preg, (const tre_char_t *)regex, n, cflags);
+109 #endif /* !TRE_WCHAR */
+110
+111 return ret;
+112 }
"regcomp.c" [Modified] The cursor is at line 112 of 188 --59%-- .
To test for possible differences between AIX 32-bit and AIX 64-bit I 
tried the following program

michael at x071:[/data/prj/cran/tests]cat sizes.c
main()
{
int s0 = sizeof((char) 0);
int s1 = sizeof(0);
int s2 = sizeof('0');
int s3 = sizeof(L'0');

printf("s1: sizeof((char) 0)\t%d\n",s0);
printf("s1: sizeof(0)\t\t%d\n",s1);
printf("s2: sizeof('0')\t\t%d\n",s2);
printf("s3: sizeof(L'0')\t%d\n",s3);
}

On POWER with gcc (Debian)

root at x066:/data/prj/cran/tests# gcc -m32 sizes.c -o sizes
sizes.c: In function ?main?:
sizes.c:8:2: warning: incompatible implicit declaration of built-in 
function ?printf? [enabled by default]
root at x066:/data/prj/cran/tests# ./sizes
s1: sizeof((char) 0) 1
s1: sizeof(0) 4
s2: sizeof('0') 4
s3: sizeof(L'0') 4
root at x066:/data/prj/cran/tests# gcc -m64 sizes.c -o sizes
sizes.c: In function ?main?:
sizes.c:8:2: warning: incompatible implicit declaration of built-in 
function ?printf? [enabled by default]
root at x066:/data/prj/cran/tests# ./sizes
s1: sizeof((char) 0) 1
s1: sizeof(0) 4
s2: sizeof('0') 4
s3: sizeof(L'0') 4

On POWER AIX and gcc:
root at x065:[/data/prj/cran/tests]gcc -maix32 -o sizes_32 sizes.c
sizes.c: In function 'main':
sizes.c:8:2: warning: incompatible implicit declaration of built-in 
function 'printf' [enabled by default]
root at x065:[/data/prj/cran/tests]gcc -maix64 -o sizes_64 sizes.c
sizes.c: In function 'main':
sizes.c:8:2: warning: incompatible implicit declaration of built-in 
function 'printf' [enabled by default]
root at x065:[/data/prj/cran/tests]./sizes_32
s1: sizeof((char) 0) 1
s1: sizeof(0) 4
s2: sizeof('0') 4
s3: sizeof(L'0') 2
root at x065:[/data/prj/cran/tests]./sizes_64
s1: sizeof((char) 0) 1
s1: sizeof(0) 4
s2: sizeof('0') 4
s3: sizeof(L'0') 4

On AIX and xlc

michael at x071:[/data/prj/cran/tests]xlc -q32 -o sizes_32xlc sizes.c
michael at x071:[/data/prj/cran/tests]./sizes_32xlc
s1: sizeof((char) 0) 1
s1: sizeof(0) 4
s2: sizeof('0') 4
s3: sizeof(L'0') 2
michael at x071:[/data/prj/cran/tests]xlc -q64 -o sizes_64xlc sizes.c
michael at x071:[/data/prj/cran/tests]./sizes_64xlc
s1: sizeof((char) 0) 1
s1: sizeof(0) 4
s2: sizeof('0') 4
s3: sizeof(L'0') 4