JDataFrame API

On Jan 15, 2016, at 12:35 PM, Thomas Fuller <thomas.fuller at coherentlogic.com> wrote:

Hi Simon,

Thanks for your feedback. -- this is an observation that I wasn't
considering when I wrote this mainly because I am, in fact, working
with rather small data sets. BTW: There is code there, it's under the
bitbucket link -- here's the direct link if you'd still like to look
at it:

https://bitbucket.org/CoherentLogic/jdataframe

Ah, sorry, all links just send you back to the page, so I missed the little filed that tells you how to check it out.

Re "for practical purposes is doesn't seem like the most efficient
solution" and "So the JSON route is very roughly ~13x slower than
using Java directly."

I've not benchmarked this and will take a closer look at what you have
today -- in fact I may include these details on the JDataFrame page.
The JDataFrame targets the use case where there's significant
development being done in Java and data is exported into R and,
additionally, the developer intends to keep the two separated as much
as possible. I could work with Java directly, but then I potentially
end up with quite a bit of Java code taking up space in R and I don't
like this because if I need to refactor something I have to do it in
two places.

No, the code is the same - it makes no difference. The R code is only one call to fetch what you need by calling your Java method. The nice thing is that you in fact save some code since there is no reason to serialize since you can simply access all Java objects directly without serialization.

There's another use case for the JDataFrame as well and that's in an
enterprise application (you may have alluded to this when you said
"[i]f you need process separation..."). Consider a business where
users are working with R and the application that produces the data is
actually running in Tomcat. Shipping large amounts of data over the
wire in this example would be a performance destroyer, but for small
data sets it certainly would be helpful from a development perspective
to expose JSON-based web services where the R script would be able to
convert a result into a data frame gracefully.

Yes, sure, that makes sense. Like I said, I would probably use some native format in that case if I worried about performance. Some candidates that come to my mind are ProtoBuf and QAP (serialization used by Rserve). If you have arrays, you can always serialize them directly which may be most efficient, but you'd probably have to write the wrapper for that yourself (annoyingly, the default Java methods use big-endian format which is slower on most machines). But then, you're right that for Tomcat applications the sizes are small enough that using JSON has the benefit that you can inspect payload by eye and/or other tools very easily.

Cheers,
Simon

On Fri, Jan 15, 2016 at 10:58 AM, Simon Urbanek
<simon.urbanek at r-project.org> wrote:

Tom,

this may be good for embedding small data sets, but for practical purposes is doesn't seem like the most efficient solution.

Since you didn't provide any code, I built a test case using the build-in Java JSON API to build a medium-sized dataset (1e6 rows) and read it in just to get a ballpark (see
https://gist.github.com/s-u/4efb284e3c15c6a2db16

# generate:
time java -cp .:javax.json-api-1.0.jar:javax.json-1.0.4.jar A > 1e6

real    0m2.764s
user    0m20.356s
sys     0m0.962s

# read:

system.time(temp <- RJSONIO::fromJSON("1e6"))

  user  system elapsed
 3.484   0.279   3.834

str(temp)

List of 2
$ V1: num [1:1000000] 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 ...
$ V2: chr [1:1000000] "X0" "X1" "X2" "X3" ...

For comparison using Java directly (includes both generation and reading into R):

system.time(temp <- lapply(J("A")$direct(), .jevalArray))

  user  system elapsed
 0.962   0.186   0.494

So the JSON route is very roughly ~13x slower than using Java directly. Obviously, this will vary by data set type etc. since there is R overhead involved as well: for example, if you have only numeric variables, the JSON route is 30x slower on reading alone [50x total]. String variables slow down everyone equally. Interestingly, the JSON encoding is using all 16 cores, so the 2.7s real time add up to over 20s CPU time so on smaller machines you may see more overhead.

If you need process separation, it may be a different story - in principle it is faster to use more native serialization than JSON since parsing is the slowest part for big datasets.

Cheers,
Simon

On Jan 14, 2016, at 4:52 PM, Thomas Fuller <thomas.fuller at coherentlogic.com> wrote:

Hi Folks,

If you need to send data from Java to R you may consider using the
JDataFrame API -- which is used to convert data into JSON which then
can be converted into a data frame in R.

Here's the project page:

https://coherentlogic.com/middleware-development/jdataframe/

and here's a partial example which demonstrates what the API looks like:

String result = new JDataFrameBuilder()
  .addColumn("Code", new Object[] {"WV", "VA", })
  .addColumn("Description", new Object[] {"West Virginia", "Virginia"})
  .toJson();

and in R script we would need to do this:

temp <- RJSONIO::fromJSON(json)
tempDF <- as.data.frame(temp)

which yields a data frame that looks like this:

tempDF

  Description Code
1 West Virginia   WV
2      Virginia   VA

It is my intention to deploy this project to Maven Central this week,
time permitting.

Questions and comments are welcomed.

Tom