Prev 44872 / 398513 Next

Computing very large distance matrix

Wed, Feb 25, 2004 5:28 PM

The problem is not with R; the problem is with Windows, as well 
as the x86 architecture.  You'll never really be able to access 
more than 3GB of physical memory (for a single process). 
Accessing more than 4GB requires a 64 bit processor.

-roger

Arnav Sheth wrote:

Hello,

Thank you all for your replies.

I have just discovered that now, for some reason, R does not let me increase
its memory limit beyond 4095 Mb.

Also, I get a different error message when computing the matrix using
daisy() as opposed to dist():

Error: Cannot allocate vector of size 135168 Kb

Does this mean that there is no way in which I can compute this distance
matrix on the machine I described below?

Is there any possible solution to this problem?

Thanks again to all,
Arnav.



----- Original Message -----
From: "Prof Brian Ripley" <ripley at stats.ox.ac.uk>
To: "Arnav Sheth" <sheth at economics.rutgers.edu>
Cc: "R-Help" <R-help at stat.math.ethz.ch>
Sent: Wednesday, February 25, 2004 2:12 PM
Subject: Re: [R] Computing very large distance matrix

On Wed, 25 Feb 2004, Arnav Sheth wrote:

Hello All,

I have a 131072x132 matrix for which I need to compute a regular

euclidean distance matrix, which I then need to transform and run agnes() on
this transformed matrix. I am having trouble computing the distance matrix
as it is fairly large and I am sure I have gone over the max.

The specific error I am getting is:
Error in vector("double", length) : negative length vectors are not

allowed

We've improved that message for the next release.

I have increased the memory limit to the maximum capacity of my hard
drive (which is around 20gb), with no success.

You'd need to increase the 32-bit limit of Windows, too ....

I am running the RGUI on Windows XP with 512 mb of RAM.

Would anyone have any suggestions as to how I can overcome this problem?
I would be most grateful for any help.

No. The distance matrix is 8Gb all by itself.

--
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

______________________________________________
R-help at stat.math.ethz.ch mailing list
https://www.stat.math.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html

Thread (6 messages)

Arnav Sheth Computing very large distance matrix Feb 25 Tom Blackwell Computing very large distance matrix Feb 25 Brian Ripley Computing very large distance matrix Feb 25 Arnav Sheth Computing very large distance matrix Feb 25 Roger D. Peng Computing very large distance matrix Feb 25 Joel Pitt Computing very large distance matrix Feb 25