Reading 64-bit integers

-----Original Message-----
From: r-devel-bounces at r-project.org 
[mailto:r-devel-bounces at r-project.org] On Behalf Of Simon Urbanek
Sent: Tuesday, March 29, 2011 6:49 PM
To: Duncan Murdoch
Cc: r-devel at r-project.org
Subject: Re: [Rd] Reading 64-bit integers


On Mar 29, 2011, at 8:47 PM, Duncan Murdoch wrote:

On 29/03/2011 7:01 PM, Jon Clayden wrote:

Dear Simon,

On 29 March 2011 22:40, Simon 

Urbanek<simon.urbanek at r-project.org>  wrote:

Jon,

On Mar 29, 2011, at 1:33 PM, Jon Clayden wrote:

Dear Simon,

Thank you for the response.

On 29 March 2011 15:06, Simon 

Urbanek<simon.urbanek at r-project.org>  wrote:

On Mar 29, 2011, at 8:46 AM, Jon Clayden wrote:

Dear all,

I see from some previous threads that support for 

64-bit integers in R

may be an aim for future versions, but in the meantime 

I'm wondering

whether it is possible to read in integers of greater 

than 32 bits at

all. Judging from ?readBin, it should be possible to 

read 8-byte

integers to some degree, but it is clearly limited in 

practice by R's

internally 32-bit integer type:

x<- as.raw(c(0,0,0,0,1,0,0,0))
(readBin(x,"integer",n=1,size=8,signed=F,endian="big"))

[1] 16777216

x<- as.raw(c(0,0,0,1,0,0,0,0))
(readBin(x,"integer",n=1,size=8,signed=F,endian="big"))

[1] 0

For values that fit into 32 bits it works fine, but 

for larger values

it fails. (I'm a bit surprised by the zero - should 

the value not be

NA if it is out of range?

No, it's not out of range - int is only 4 bytes so only 

4 first bytes (respecting endianness order, hence LSB) are used.

The fact remains that I ask for the value of an 8-byte 

integer and

don't get it.

I think you're misinterpreting the documentation:

   If 'size' is specified and not the natural size of the object,
   each element of the vector is coerced to an appropriate type
   before being written or as it is read.

The "integer" object type is defined as signed 32-bit in 

R, so if you ask for "8 bytes into object type integer", you 
get a coercion into that object type -- 32-bit signed integer 
-- as documented. I think the issue may come from the 
confusion of the object type "integer" with general "integer 
number" in mathematical sense that has no representation 
restrictions. (FWIW in C the "integer" type is "int" and it 
is 32-bit on all modern OSes regardless of platform - that's 
where the limitation comes from, it's not something R has made up).

OK, but it still seems like there is a case for raising a 

warning. As

it is there is no way to tell when reading an 8-byte integer from a
file whether its value is really 0, or if it merely has 0 in its
least-significant 4 bytes. If 99% of such stored numbers are below
2^31, one is going to need some extra logic to catch the other 1%
where you (silently) get the wrong value. In essence, unless you're
certain that you will never come across a number that actually uses
the upper 4 bytes, you will always have to read it as two 4-byte
numbers and check that the high-order one (which is endianness
dependent, of course) is zero. A C-level sanity check seems more
efficient and more helpful to me.

Seems to me that the S-PLUS solution (output="double") 

would be a lot more useful.  I'd commit that if you write it; 
I don't think I'd commit the warning.

I was going to write some thing similar (idea = good, patch 
welcome ;)). My only worry is that the "output" argument is a 
bit misleading in that one could expect to use any 
combination of "input"/"output" which may be a maintenance 
nightmare. If I understand it correctly it's only a special 
case for integer input. I don't have S+ so can't say how they 
deal with that.

In S+'s readBin the output argument can be
only double() or single() when what is double()
or single() (S+ still  has a real single
precision storage mode) and can be any
numeric type or logical when what is integer().

The output=double() seemed like the only useful case.

It does not warn when precision is lost in the 8-byte
integer to double conversion.  Perhaps it should.

Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com  

Cheers,
Simon

Pretending that it's really only four bytes because of
the limits of R's integer type isn't all that helpful. Perhaps a
warning should be put out if the cast will affect the 

value of the

result? It looks like the relevant lines in 

src/main/connections.c are

3689-3697 in the current alpha:

#if SIZEOF_LONG == 8
                 case sizeof(long):
                     INTEGER(ans)[i] = (int)*((long *)buf);
                     break;
#elif SIZEOF_LONG_LONG == 8
                 case sizeof(_lli_t):
                     INTEGER(ans)[i] = (int)*((_lli_t *)buf);
                     break;
#endif

) The value can be represented as a double,
though:

4294967296

[1] 4294967296

I wouldn't expect readBin() to return a double if an 

integer was

requested, but is there any way to get the correct 

value out of it?

Trivially (for your unsigned big-endian case):

y<- readBin(x, "integer", n=length(x)/4L, endian="big")
y<- ifelse(y<  0, 2^32 + y, y)
i<- seq(1,length(y),2)
y<- y[i] * 2^32 + y[i + 1L]

Thanks for the code, but I'm not sure I would call that trivial,
especially if one needs to cater for little endian and 

signed cases as

well!

I was saying for your case and it's trivial as in read as 

integers, convert to double precision and add.

This is what I meant by reconstructing the number manually...

You didn't say so - you were talking about reconstructing 

it from a raw vector which seems a lot more painful since you 
can't compute with enough precision on raw vectors.

True - I should have been more specific. Sorry.

Jon