R, AIX 64-bit builds - trying to understand root cause for message: "Error: Line starting 'Package: tools ...' is malformed!"
On 04-Jan-16 15:52, Simon Urbanek wrote:
On Jan 4, 2016, at 1:34 AM, Michael Felt <aixtools at gmail.com> wrote:
I would be "pleased" if you would try packages - i.e., none of "Toolbox" or Perzl rpm's.
R requires iconv, so you have to get that somehow. I tried to bypass the check and build it anyway against the system one, but that fails as there is a iconv conversion further down the line that is not supported. Building iconv is a PITA because it conflicts with the system one so other tools break when you modify the load path. Perzl's iconv fixes that by incorporating both the system objects and the GNU ones into the same binary and that seems to work reliably so far. That is the only dependency I was using (his iconv doesn't have any dependencies itself).
I would have to install another system and install perzl iconv -
however, I assume that it also sits in /usr/lib (and/or has a symbolic
link to it). I have mine installed in /opt/lib - so the default shared
libraries look like:
cran at x065:[/home/cran/64/R]ldd /home/cran/64/R/bin/exec/R
/home/cran/64/R/bin/exec/R needs:
/usr/lib/libc.a(shr_64.o)
/usr/lib/libpthread.a(shr_xpg5_64.o)
/opt/lib/libintl.a(libintl.so.8)
/opt/lib/libiconv.a(libiconv.so.2)
/unix
/usr/lib/libcrypt.a(shr_64.o)
/usr/lib/libpthread.a(shr_xpg5.o)
/usr/lib/libc.a(shr.o)
/usr/lib/libpthreads.a(shr_comm.o)
/usr/lib/libcrypt.a(shr.o)
I have tested with adding my (gnu iconv) to the /usr/lib archive, as the
search does seem to get corrupted.
For testing purposes I have added the 64-bit member to my 32-bit
packaging - rather than a seperate archive in /opt/lib64
cran at x065:[/home/cran/64/R]ar -X32_64 -tv /opt/lib/libiconv.a
rwxr-xr-x 0/0 1010404 Nov 27 17:28 2014 libiconv.so.2
rwxr-xr-x 0/0 111901 Nov 27 17:28 2014 shr4.o
rwxr-xr-x 0/0 112151 Nov 27 17:28 2014 shr.o
rwxr-xr-x 0/0 1035470 Dec 3 15:11 2015 libiconv.so.2
I did something similiar to this for all the dependencies earlier when I
was working with R-devel branch. When the build is finishing I will
workout how to install both into a single archive (the hard part is a
clean uninstall afterwards, and harder is managing an update. So,
ideally, I will go to having one package with both 32 and 64 bit members.)
The iconv I supply might not be working as is (I will need to install a new system to check). However, this is why I have been testing with R-3.2.3 - to side-step the system library dependencies. re: xz - I have "installp" version - and any of my packages should work side-by-side with the Toolbox/.rpm files - my PATH prefix is /opt (so, /opt/bin and /opt/sbin rather than /usr/bin, /usr/local/bin (etc.).
It fails even with the built-in one in R so it has nothing to do with the xz used itself.
What is the built-in level - do not know off the top of my head. Perzl is 5.0.7 and I have been using 5.0.8 (when working with R-devel).
I am testing on a Power6, usually with 2 to 4G RAM active. When I get the compile to finish I will up to 8G to test some data processing that fails with the 32-bit limitations. Another question: are you also getting lots of duplicate symbol warnings? Curious me. If you are I would like to send my modifications to configure(.ac) for you to test (I put .ac in () because I am actually editting configure atm but shall put them into configure.ac when I finish)
No, no duplicate warnings.
Then I suspect the duplicate warnings have to do with how the m4 macros are processing the gcc -v flags (actually, the gfortran -v flags). Adding full path to libgcc.a and including -bexpall causes symbols to be imported and reexported - hence duplicate. Another reason for me to get xlfortran installed and "test" against that as well.
FYI: I shall be downloading the "try and buy" xlc and xlfortran - and I think you will certainly prefer my packaging then as they work without the libgc dependencies that many of the rpm packages need. And, at your option - we can take this discussion over tools - "out of forums". Or at least start a new thread.
We could leave the list out and create a Wiki or something with the results of our tests.
I would be happy to sponsor that - either on my forums (http://forums.rootvg.net/aixtools/r-for-aix/) or I can look into how I create a userid for you on my aixtools wiki. I started something in November http://www.aixtools.net/index.php/Projects#%22Work%20in%20Progress%22, but basically, am waiting for better results before dedicating a page to R. In short, I would prefer the forums - as anyone could join the discussion without any intervention from me.
re: unsigned short - I have adopted the convention to use the *NN_t types after running into a problem with unsigned longlong (from a very old program). It goes back many years - and maybe they have finalized short to mean 16-bits - but I remember when short was meant to help when moving from 16-bit to 32-bit and the "word" size changed - i.e., int became 32-bit same as long. nd for a long (no pun intended) long was 32-bit and long long was 64-bit. Those definitions are extinct.
I'm not sure what you refer here - the issue with TRE has nothing to do with short - it can take any int type and like I said most platforms use unsigned int which is big enough on all platforms.
I claim no expertise. My observation is that UTF-XX seems to be 16-bit. I say seems, because there may be a multi-byte requirement that does not fit in 16-bits and then a larger size is appropriate. From my perspective (as a porter) it is easier to identify inconsistency when typedef definitions are not directly related to short, int or long. In binary it may not make any difference as far as sizeof() is concerned. I just hope for improved legibility. I am not "r-project" - and my assumption is that "r-project" does the commits. My contribution is merely my opinion. What you decide is what counts - not my opinion.
imho - the standard for wint_t is wrong as well - based on an assumption about how "short" is defined. And, I consider it poor practice that there are som many cases of type cast switches between ushort and int.
Not really - it doesn't care about short at all - note that the short typedef is never actually used on AIX since it has wchar support so TRE is only using int.
You lost me here. I am aware that AIX has a very different set of
include files that tre is not using. My perspective is that R is
ignoring/overrulling any IBM/AIX wchar processing. See my "not an
expert" defense above.
Mainly - I am happy that "make sysdata" is working.
p.s. I shall make a new entry on my forums re: where I am at now re:
Warning in solve.default(rgb) :
unable to load shared object '/home/cran/64/R/modules//lapack.so':
rtld: 0712-001 Symbol __powidf2 was referenced
from module /home/cran/64/R/lib/libRlapack.so(), but a runtime
definition
of the symbol was not found.
rtld: 0712-001 Symbol _gfortran_pow_i4_i4 was referenced
from module /home/cran/64/R/lib/libRlapack.so(), but a runtime
definition
of the symbol was not found.
rtld: 0712-001 Symbol
_gfortran_compare_string/home/cran/64/R/lib/libRlapack.so was referenced
from module /home/cran/64/R/lib/libRlapack.so(), but a runtime
definition
of the symbol was not found.
rtld: 0712-001 Symbol _gfortran_concat_string was referenced
from module /home/cran/64/R/lib/libRlapack.so(), but a runtime
definition
of the symbol was not found.
rtld: 0712-002 fatal error: exiting.
Error in solve.default(rgb) : LAPACK routines cannot be loaded
Error: unable to load R code in package 'grDevices'
Cheers, Simon
Michael On 04-Jan-16 01:26, Simon Urbanek wrote:
Michael, I'm using xlc + xlf - the exact flags are configure CC=xlc_r CXX=xlc++_r F77=xlf_r FC=xlf95_r LIBS='-L/opt/freeware/lib /opt/freeware/lib/libiconv.a -lpthread' --prefix=/opt/freeware CPPFLAGS=-I/opt/freeware/include with export OBJECT_MODE=64 in the env (and libiconv from perzl.org RPMs - iconv is a serious pain). I have changed the TRE typedef from "wint_t" to "unsigned int" since that is what the other platforms use anyway. On my AIX7 VM I'm running out of memory in xz when lazy-loading is enabled which his odd - the VM has 4Gb which one would think should be enough. The symptom is that installing MASS fails when creating lazy-load rds or the same happens when running make check as memCompress() test for xz fails. I wish there was a way to disable xz altogether.. I'd like to get IBM compilers to work first since those are the official ones. I may try gcc later, but I'm really interested in the IBM ones because that gives us a good non-GNU test (e.g., TRE fails in JIT due some alleged GNUisms so you can't use the strict compiler mode). Cheers, Simon On Jan 3, 2016, at 10:59 AM, Michael Felt <aixtools at gmail.com> wrote:
On 2016-01-01 23:48, peter dalgaard wrote:
Nice catch you two!!! Happy New Year -pd
I am much happier with this great start! Simon - which compiler)s) did you use: xlc and xlfortran, or gcc/gfortran? I have made some changes to configure(.ac) so maybe my problems are self-inflicted. But would be good to know what environment you are using. Thanks for looking - and finding!!! Michael
On 01 Jan 2016, at 22:06 , Simon Urbanek<simon.urbanek at r-project.org> wrote: Ok, found the problem - on platforms that support it TRE uses wint_t (from wchar.h) as its type for characters (tre_cint_t) which on AIX is *signed* int. TRE uses liberally conversions between int and tre_cint_t apparently assuming that the latter is unsigned so conversions back to int are suitable for comparisons etc. On other platforms wint_t is unsigned so it works. Manually defining tre_cint_t to unsigned int fixes the issue. Cheers, Simon On Jan 1, 2016, at 12:20 PM, Simon Urbanek<simon.urbanek at r-project.org> wrote:
Michael, thanks, I'll have a look once my PDP VMs are up again (later today). This may be a signedness issue although it's unclear why other platforms wouldn't be affected. Cheers, Simon On Dec 31, 2015, at 10:14 AM, Michael Felt<aixtools at gmail.com> wrote:
On 2015-12-30 09:58, Michael Felt wrote:
On 2015-12-29 11:02, Michael Felt wrote:
This seems to be a problem that goes back a long time - and I hope someone who understands what tre is suppossed to be doing will look at this. A short history of other people who have reported on this on different versions of AIX. I shall only add that I get the same results on AIX 5.3 TL7, AIX 6.1 TL9 and AIX 7.1 TL3. Basically, with settings that work for AIX and 32-bit - the only changes being -maix32 becomes -maix64 and export OBJECT_MODE=32 becomes export OBJECT_MODE=64 Then to shorten the 'make' bla bla, first run just make, then cd src/library/tools make -s sysdata http://article.gmane.org/gmane.comp.lang.r.devel/38817/match=package+tools+malformed http://article.gmane.org/gmane.comp.lang.r.devel/36886/match=package+tools+malformed http://article.gmane.org/gmane.comp.lang.r.devel/23372/match=package+tools+malformed Date: 2010-01-25 06:55:41 GMT (5 years, 48 weeks, 1 day, 20 hours and 30 minutes ago) To that, to get debug data, I have * added -DTRE_DUGUG to src/extra/tre/Makefile # ALL_CFLAGS = $(ALL_CFLAGS_LO) -DTRE_DEBUG * rm src/extra/tre/tre-match-parallel.o * find . -name \*.so -exec rm {} \; * make * cd src/library/tools * make -s sysdata Attached are the two script files of the screen output. The 32-bit one is more verbose - and contains magically lines such as: found match 3037fd14 (while "found" does not occur in the 64-bit output) root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]wc /tmp/sysdata.??.* 4730 14123 139916 /tmp/sysdata.32.text 1312 3688 40528 /tmp/sysdata.64.text 6042 17811 180444 total root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]grep -c found /tmp/sysdata.??.* /tmp/sysdata.32.text:19 /tmp/sysdata.64.text:0 Hope this brings us (or me), closer to a resolution to an old concern. And, best wishes for the new year! Michael
Still hoping for someones curiosity/willingness. The differences show up in the first comparision that is made (of the string "3.2.3" it seems) - 32-bit is on the left, 64-bit on the right. Script command is started on Tue Dec 29 08:39:16 UTC 2015. | Script command is started on Tue Dec 29 08:39:56 UTC 2015. root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make -s sysdata | root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make -s sysdata installing 'sysdata.rda' | installing 'sysdata.rda' tre_tnfa_run_parallel, input type 1 | tre_tnfa_run_parallel, input type 1 length: -1 | length: -1 pos:chr/code | states and tags | pos:chr/code | states and tags -------------+------------------------------------------------ | -------------+------------------------------------------------ init> 30380200 3038014c 30380098 | init> 110cc3040 110cc2f28 110cc2e10 match end offset = -1 | match end offset = -1 tre_tnfa_run_parallel, input type 1 | tre_tnfa_run_parallel, input type 1 length: -1 | length: -1 pos:chr/code | states and tags | pos:chr/code | states and tags -------------+------------------------------------------------ | -------------+------------------------------------------------ init> 3037fb88 | init> 110cc3310 0: 3/00051 | 3037fb88/0:0 | 0: 3/00051 | 110cc3310/0:0 1: ./00046 | 3037fb88/0:0 | 1: ./00046 | 110cc3310/0:0 init> 3037fb88 | init> 110cc3310 1: ./00046 | 3037fb88/0:1 | 1: ./00046 | 110cc3310/0:1 2: 2/00050 | 3037fb88/0:1 | 2: 2/00050 | 110cc3310/0:1 assertion failed | assertion failed init> 3037fb88 | init> 110cc3310 2: 2/00050 | 3037fc18/0:1 3037fb88/0:2 | 2: 2/00050 | 110cc33f0/0:1 110cc3310/0:2 3: ./00046 | 3037fc18/0:1 3037fb88/0:2 | 3: ./00046 | 110cc33f0/0:1 110cc3310/0:2 assertion failed *** DIFFERENCE *** | init> 110cc3310 init> 3037fb88 | 3: ./00046 | 110cc3310/0:3 3: ./00046 | 3037fc18/0:1 3037fb88/0:3 | 4: 3/00051 | 110cc3310/0:3 4: 3/00051 | 3037fc18/0:1 3037fb88/0:3 | assertion failed assertion failed | init> 110cc3310 init> 3037fb88 | 4: 3/00051 | 110cc33f0/0:3 110cc3310/0:4 4: 3/00051 | 3037fc18/0:3 3037fb88/0:4 | 5: /00000 | 110cc33f0/0:3 110cc3310/0:4 5: /00000 | 3037fc18/0:3 3037fb88/0:4 | init> 110cc3310 found match 3037fd14 *** DIFFERENCE *** | match end offset = -1 match end offset = 5 *** DIFFERENCE *** | tre_tnfa_run_parallel, input type 1 tre_tnfa_run_parallel, input type 1 | length: -1 length: -1 | pos:chr/code | states and tags pos:chr/code | states and tags | -------------+------------------------------------------------ -------------+------------------------------------------------ | init> 110cc4780 110cc4668 110cc4550 init> 303811c0 3038110c 30381058 | match end offset = -1 match end offset = -1 | tre_tnfa_run_parallel, input type 1 tre_tnfa_run_parallel, input type 1 | length: -1 length: -1 | pos:chr/code | states and tags pos:chr/code | states and tags | -------------+------------------------------------------------ -------------+------------------------------------------------ | init> 110cc5700 110cc55e8 110cc54d0
One day further - looks like tre_compile (or just before, after all).
With TRE_DEBUG switched on in tre-compile.c and tre-ast.c I see (snip)
--- /tmp/x.32 2015-12-31 15:09:44.000000000 +0000
+++ /tmp/x.64 2015-12-31 15:09:30.000000000 +0000
@@ -1,5 +1,5 @@
- Script command is started on Thu Dec 31 15:04:39 2015.
- root at x069:[/data/prj/cran/32/R-aix-3.2.3/src/library/tools]make sysdata
+ Script command is started on Thu Dec 31 15:08:43 2015.
+ root at x069:[/data/prj/cran/64/R-aix-3.2.3/src/library/tools]make sysdata
installing 'sysdata.rda'
echo "tools:::sysdata2LazyLoadDB(\"/data/prj/cran/R-3.2.3/src/library/tools/R/sysdata.rda\",\"../../../library/tools/R\")" | \
R_DEFAULT_PACKAGES=NULL LC_ALL=C ../../../bin/R --vanilla --slave
@@ -167,7 +167,7 @@
initial: 1/1,0, assert 0
initial: 0/0, assert 0
initial: 0/0, assert 0
- final state 30370718
+ final state 110cba530
tre_compile: parsing '(^|[^%])(%%)*%V'
AST:
catenation, sub 0, 0 tags
@@ -177,7 +177,7 @@
assertions: bol
union, sub -1, 0 tags
literal (, $) (0, 36), pos 0, sub -1, 0 tags
- literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags
+ literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags
iteration {0, -1}, sub -1, 0 tags, greedy
catenation, sub 2, 0 tags
literal (%, %) (37, 37), pos 1, sub -1, 0 tags
@@ -197,7 +197,7 @@
Union
Literal 0-36
After union left
- Literal 38-65535
+ Literal 38--1
After union right
After union right
num_tags += 2
@@ -231,7 +231,7 @@
assertions: bol
union, sub -1, 0 tags
literal (, $) (0, 36), pos 0, sub -1, 0 tags
- literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags
+ literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags
iteration {0, -1}, sub -1, 2 tags, greedy
catenation, sub 2, 1 tags
literal (%, %) (37, 37), pos 1, sub -1, 1 tags
@@ -255,7 +255,7 @@
Union
Literal 0-36
After union left
- Literal 38-65535
+ Literal 38--1
After union right
After union right
tre_add_tag_right: tag 3
@@ -342,7 +342,7 @@
catenation, sub -1, 0 tags
union, sub -1, 0 tags
literal (, $) (0, 36), pos 0, sub -1, 0 tags
- literal (&, M-^?) (38, 65535), pos 0, sub -1, 0 tags
+ literal (&, M-^?) (38, -1), pos 0, sub -1, 0 tags
tag 4
It seems in 32-bit mode -1 is unsigned (65535) but -1 == -1 in 64-bit mode.
I suspect I will "find it" - but a proposed change is appreciated.
Happy New Year,
Michael
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________ R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel