Message-ID: <20090324044104.vvgggeyjwgcgscwo@webmail3.isis.unc.edu>
Date: 2009-03-24T08:41:04Z
From: Guillaume Filteau
Subject: How to separate huge dataset into chunks
Hello all,
I?m trying to take a huge dataset (1.5 GB) and separate it into smaller
chunks with R.
So far I had nothing but problems.
I cannot load the whole dataset in R due to memory problems. So, I
instead try to load a few (100000) lines at a time (with read.table).
However, R kept crashing (with no error message) at about the 6800000
line. This is extremely frustrating.
To try to fix this, I used connections with read.table. However, I now
get a cryptic error telling me ?no lines available in input?.
Is there any way to make this work?
Best,
Guillaume