An embedded and charset-unspecified text was scrubbed... Name: not available URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20130730/e68f87b9/attachment.pl>
xmlToDataFrame very slow
2 messages · Stavros Macrakis, Duncan Temple Lang
Hi Stavros xmlToDataFrame() is very generic and so doesn't know anything about the particulars of the XML it is processing. If you know something about the structure of the XML, you should be able to leverage that for performance. xmlToDataFrame is also not optimized as it is just a convenience routine for people who want to work with XML without much effort. If you send me the file and the code you are using to read the file, I'll take a look at it. D.
On 7/30/13 11:10 AM, Stavros Macrakis wrote:
I have a modest-size XML file (52MB) in a format suited to xmlToDataFrame (package XML).
I have successfully read it into R by splitting the file 10 ways then running xmlToDataFrame on each part, then
rbind.fill (package plyr) on the result. This takes about 530 s total, and results in a data.frame with 71k rows and
object.size of 21MB.
But trying to run xmlToDataFrame on the whole file takes forever (> 10000 s so far). xmlParse of this file takes only 0.8 s.
I tried running xmlToDataFrame on the first 10% of the file, then the first 10% repeated twice, then three times (with
the outer tags adjusted of course). Timings:
1 copy: 111 s = 111 per copy
2 copy: 311 s = 155 " "
3 copy: 626 s = 209 " "
The runtime is superlinear. What is going on here? Is there a better approach?
Thanks,
-s