Message-ID: <gu5cvt$ok3$1@ger.gmane.org>
Date: 2009-05-10T02:09:00Z
From: Rob Steele
Subject: Reading large files quickly
In-Reply-To: <gu4apd$en0$1@ger.gmane.org>
Thanks guys, good suggestions. To clarify, I'm running on a fast
multi-core server with 16 GB RAM under 64 bit CentOS 5 and R 2.8.1.
Paging shouldn't be an issue since I'm reading in chunks and not trying
to store the whole file in memory at once. Thanks again.
Rob Steele wrote:
> I'm finding that readLines() and read.fwf() take nearly two hours to
> work through a 3.5 GB file, even when reading in large (100 MB) chunks.
> The unix command wc by contrast processes the same file in three
> minutes. Is there a faster way to read files in R?
>
> Thanks!
>