Please make Pre-3.1 read.csv (type.convert) behavior available

Hi,

We at 0xdata use Java and R together, and the new behavior for read.csv has
made R unable to read the output of Java?s Double.toString().

This, needless to say, is disruptive for us.  (Actually, it was downright shocking.)

+1 for restoring old behavior.

On 26 April 2014 at 07:28, Duncan Murdoch wrote:
| On 26/04/2014, 12:23 AM, Tom Kraljevic wrote:
| >
| > Hi,
| >
| > We at 0xdata use Java and R together, and the new behavior for read.csv has
| > made R unable to read the output of Java?s Double.toString().
| 
| It may be less convenient, but it's certainly not "unable".  Use colClasses.
| 
| 
| >
| > This, needless to say, is disruptive for us.  (Actually, it was downright shocking.)
| 
| It wouldn't have been a shock if you had tested pre-release versions. 
| Commercial users of R should be contributing to its development, and 
| that's a really easy way to do so.

Seconded. For what it is worth, I made five pre-release available within
Debian. Testing thses each was just an apt-get away.

In any event, you can also farm out the old behaviour to a (local or even
CRAN) package that provides the old behaviour if your life depends upon it.

Or you could real serialization rather than relying on the crutch that is csv.

Dirk

-- 
Dirk Eddelbuettel | edd at debian.org | http://dirk.eddelbuettel.com

Hi Duncan,


Please allow me to add a bit more context, which I probably should have
added to my original message.

We actually did see this in an R 3.1 beta which was pulled by an apt-get
and thought it had been released
accidentally.  From my user perspective, the parsing of a string like
?1.2345678901234567890? into a
factor was so surprising, I actually assumed it was just a really bad
bug that would be fixed before the
?real" release.  I didn?t bother reporting it since I assumed beta users
would be heavily impacted and
there is no way it wouldn?t be fixed.  Apologies for that mistake on my
part.

After discovering this new behavior really got released GA, I went
searching to see what was going on.
I found this bug, which states ?If you wish to express your opinion
about the new behavior, please do so
on the R-devel mailing list."

https://bugs.r-project.org/bugzilla/show_bug.cgi?id=15751

So I?m sharing my opinion, as suggested.  Thanks to all for the time
spent reading my opinion.


Let me also say, we are huge fans of R; many of our customers use R, and
we greatly appreciate the
efforts of the R core team.  We are in the process of contributing an
H2O package back to the R
community and thanks to the CRAN moderators, as well, for their
assistance in this process.
CRAN is a fantastic resource.


I would like to share a little more insight on how this behavior affects
us, in particular.  These merits
have probably already been debated, but let me state them here again to
provide the appropriate
context.

1.  When dealing with larger and larger data, things become cumbersome.
  Your comment that
specifying column types would work is true.  But when there are
thousands+ of columns, specifying
them one by one becomes more and more of a burden, and it becomes easier
to make a mistake.
And when you do make a mistake, you can imagine a tool writer choosing
to just ?do what it?s told?
and swallowing the mistake.  (Trying not to be smarter than the user.)

2.  When working with datasets that have more and more rows, sometimes
there is a bad row.
Big data is messy.  Having one bad value in one bad row contaminate the
entire dataset can be
undesirable for some.  When you have millions of rows or more, each row
becomes less precious.
Many people would rather just ignore the effects of the bad row than try
to fix it.  Especially in this
case, when ?bad? means a bit of extra precision that likely won?t have a
negative impact on the result.
(In our case, this extra precision was the output of Java?s
Double.toString().)

Our users want to use R as a driver language and a reference tool.
  Being able to interchange
data easily (even just snippets) between tools is very valuable.


Thanks,
Tom


Below is an example of how you can create a million row dataset which
works fine (parses as a
numeric), but then adding just one bad row (which still *looks*
numeric!) flips the entire column to
a factor.  Finding that one row out of a million+ can be quite a challenge.


# Script to generate dataset.
$ cat genDataset.py
#!/usr/bin/env python

for x in range(0, 1000000):
     print (str(x) + ".1")

# Generate the dataset.
$ ./genDataset.py > million.csv

# R 3.1 thinks it?s a numeric.
$ R

 > df = read.csv("million.csv")
 > str(df)

'data.frame':999999 obs. of  1 variable:
  $ X0.1: num  1.1 2.1 3.1 4.1 5.1 6.1 7.1 8.1 9.1 10.1 ...

# Add one more over-precision row.
$ echo "1.2345678901234567890" >> million.csv

# Now R 3.1 thinks it?s a factor.
$ R

 > df2 = read.csv("million.csv")
 > str(df2)

'data.frame':1000000 obs. of  1 variable:
  $ X0.1: Factor w/ 1000000 levels "1.1","1.2345678901234567890",..: 1
111113 222224 333335 444446 555557 666668 777779 888890 3 ...





On Apr 26, 2014, at 4:28 AM, Duncan Murdoch <murdoch.duncan at gmail.com
<mailto:murdoch.duncan at gmail.com>> wrote:

On 26/04/2014, 12:23 AM, Tom Kraljevic wrote:

Hi,

We at 0xdata use Java and R together, and the new behavior for
read.csv has
made R unable to read the output of Java?s Double.toString().

It may be less convenient, but it's certainly not "unable".  Use
colClasses.

This, needless to say, is disruptive for us.  (Actually, it was
downright shocking.)

It wouldn't have been a shock if you had tested pre-release versions.
Commercial users of R should be contributing to its development, and
that's a really easy way to do so.

Duncan Murdoch

+1 for restoring old behavior.

bad.df = read.csv(file="/Users/tomk/bad.csv", header=F)
str(bad.df)

bad.df = read.csv(file="/Users/tomk/bad.csv", header=F)
str(bad.df)

bad.df = read.csv(file="/Users/tomk/bad.csv", header=F)
str(bad.df)

bad.df = read.csv(file="/Users/tomk/bad.csv", header=F)
str(bad.df)

bad.df = read.csv(file="/Users/tomk/bad.csv", header=F)
str(bad.df)

bad.df = read.csv(file="/Users/tomk/bad.csv", header=F)
str(bad.df)

bad.df = read.csv(file="/Users/tomk/bad.csv", header=F)
str(bad.df)

Hi,


One additional follow-up here.

Unfortunately, I hit what looks like an R parsing bug that makes the Java Double.toHexString() output
unreliable for reading by R.  (This is really unfortunate, because the format is intended to be lossless
and it looks like it?s so close to fully working.)

You can see the spec for the conversion here:
     http://docs.oracle.com/javase/7/docs/api/java/lang/Double.html#toHexString(double)

The last value in the list below is not parsed by R in the way I expected, and causes the column to flip
from numeric to factor.


-0x1.8ff831c7ffffdp-1
-0x1.aff831c7ffffdp-1
-0x1.bff831c7ffffdp-1
-0x1.cff831c7ffffdp-1
-0x1.dff831c7ffffdp-1
-0x1.eff831c7ffffdp-1
-0x1.fff831c7ffffdp-1           <<<<< this value is not parsed as a number and flips the column from numeric to factor.

Below is the R output from adding one row at a time to ?bad.csv?.
The last attempt results in a factor rather than a numeric column.

What?s really odd about it is that the .a through .e case work fine but the .f case doesn?t.


Thanks,
Tom

bad.df = read.csv(file="/Users/tomk/bad.csv", header=F)
str(bad.df)

'data.frame':	1 obs. of  1 variable:
  $ V1: num -0.781

bad.df = read.csv(file="/Users/tomk/bad.csv", header=F)
str(bad.df)

'data.frame':	2 obs. of  1 variable:
  $ V1: num  -0.781 -0.844

bad.df = read.csv(file="/Users/tomk/bad.csv", header=F)
str(bad.df)

'data.frame':	3 obs. of  1 variable:
  $ V1: num  -0.781 -0.844 -0.875

bad.df = read.csv(file="/Users/tomk/bad.csv", header=F)
str(bad.df)

'data.frame':	4 obs. of  1 variable:
  $ V1: num  -0.781 -0.844 -0.875 -0.906

bad.df = read.csv(file="/Users/tomk/bad.csv", header=F)
str(bad.df)

'data.frame':	5 obs. of  1 variable:
  $ V1: num  -0.781 -0.844 -0.875 -0.906 -0.937

bad.df = read.csv(file="/Users/tomk/bad.csv", header=F)
str(bad.df)

'data.frame':	6 obs. of  1 variable:
  $ V1: num  -0.781 -0.844 -0.875 -0.906 -0.937 ...

bad.df = read.csv(file="/Users/tomk/bad.csv", header=F)
str(bad.df)

'data.frame':	7 obs. of  1 variable:
  $ V1: Factor w/ 7 levels "-0x1.8ff831c7ffffdp-1",..: 1 2 3 4 5 6 7

Hi Duncan,


This program and output should answer your question regarding java behavior.

Basically the character toHexString() representation is shown to be
lossless for this
example (in Java).

Please let me know if there is any way I can help further.  I?d love for
this to work!
I would be happy to put all this into an R bug report if that is
convenient for you.

Thanks,
Tom




$ cat example.java
class example {
     public static void main(String[] args) {
         String value_as_string = "-0x1.fff831c7ffffdp-1";
         double value = Double.parseDouble(value_as_string);
         System.out.println("Starting string    : " + value_as_string);
         System.out.println("value toString()   : " +
Double.toString(value));
         System.out.println("value toHexString(): " +
Double.toHexString(value));

         long bits = Double.doubleToRawLongBits(value);
         boolean isNegative = (bits & 0x8000000000000000L) != 0;
         long biased_exponent      = (bits & 0x7ff0000000000000L) >> 52;
         long exponent = biased_exponent - 1023;
         long mantissa =  bits & 0x000fffffffffffffL;
         System.out.println("isNegative         : " + isNegative);
         System.out.println("biased exponent    : " + biased_exponent);
         System.out.println("exponent           : " + exponent);
         System.out.println("mantissa           : " + mantissa);
         System.out.println("mantissa as hex    : " +
Long.toHexString(mantissa));
     }
}


$ javac example.java
$ java example
Starting string    : -0x1.fff831c7ffffdp-1
value toString()   : -0.999940448440611
value toHexString(): -0x1.fff831c7ffffdp-1
isNegative         : true
biased exponent    : 1022
exponent           : -1
mantissa           : 4503063234609149
mantissa as hex    : fff831c7ffffd


$ java -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)



On Apr 26, 2014, at 2:18 PM, Duncan Murdoch <murdoch.duncan at gmail.com
<mailto:murdoch.duncan at gmail.com>> wrote:

On 26/04/2014, 4:12 PM, Tom Kraljevic wrote:

Hi,


One additional follow-up here.

Unfortunately, I hit what looks like an R parsing bug that makes the
Java Double.toHexString() output
unreliable for reading by R.  (This is really unfortunate, because
the format is intended to be lossless
and it looks like it?s so close to fully working.)

You can see the spec for the conversion here:
http://docs.oracle.com/javase/7/docs/api/java/lang/Double.html#toHexString(double)
<http://docs.oracle.com/javase/7/docs/api/java/lang/Double.html#toHexString%28double%29>

The last value in the list below is not parsed by R in the way I
expected, and causes the column to flip
from numeric to factor.


-0x1.8ff831c7ffffdp-1
-0x1.aff831c7ffffdp-1
-0x1.bff831c7ffffdp-1
-0x1.cff831c7ffffdp-1
-0x1.dff831c7ffffdp-1
-0x1.eff831c7ffffdp-1
-0x1.fff831c7ffffdp-1           <<<<< this value is not parsed as a
number and flips the column from numeric to factor.

That looks like a bug in the conversion code.  It uses the same test
for lack of accuracy for hex doubles as it uses for decimal ones, but
hex doubles can be larger before they lose precision.  I believe the
largest integer that can be represented exactly is 2^53 - 1, i.e.

0x1.fffffffffffffp52

in this notation; can you confirm that your Java code reads it and
writes the same string?  This is about 1% bigger than the limit at
which type.convert switches to strings or factors.

Duncan Murdoch

df = read.csv("bad1.csv", header=F)
str(df)

df = read.csv("bad1.csv", header=F)
str(df)

 df = read.csv("bad1.csv", header=F)
str(df)

This, needless to say, is disruptive for us.  (Actually, it was downright shocking.)