An embedded and charset-unspecified text was scrubbed... Name: not available Url: https://stat.ethz.ch/pipermail/r-help/attachments/20040225/71bb1fb1/attachment.pl
Computing very large distance matrix
6 messages · Arnav Sheth, Tom Blackwell, Brian Ripley +2 more
Arnav - A suggestion I have made in the past is to run Chris Fraley and Adrian Raftery's mclust() procedure instead of Rousseuw's agnes(), if you are willing to use a different clustering method. The pdf instruction manual for the mclust package includes explicit suggestions for how to use mclust with large data sets. Naive attempts are guaranteed to fail, so DO read the manual. - tom blackwell - u michigan medical school - ann arbor -
On Wed, 25 Feb 2004, Arnav Sheth wrote:
Hello All,
I have a 131072x132 matrix for which I need to compute a regular euclidean distance matrix, which I then need to transform and run agnes() on this transformed matrix. I am having trouble computing the distance matrix as it is fairly large and I am sure I have gone over the max.
The specific error I am getting is:
Error in vector("double", length) : negative length vectors are not allowed
I have increased the memory limit to the maximum capacity of my hard drive (which is around 20gb), with no success.
I am running the RGUI on Windows XP with 512 mb of RAM.
Would anyone have any suggestions as to how I can overcome this problem? I would be most grateful for any help.
Thanks,
Arnav
[[alternative HTML version deleted]]
______________________________________________ R-help at stat.math.ethz.ch mailing list https://www.stat.math.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
On Wed, 25 Feb 2004, Arnav Sheth wrote:
Hello All,
I have a 131072x132 matrix for which I need to compute a regular euclidean distance matrix, which I then need to transform and run agnes() on this transformed matrix. I am having trouble computing the distance matrix as it is fairly large and I am sure I have gone over the max.
The specific error I am getting is:
Error in vector("double", length) : negative length vectors are not allowed
We've improved that message for the next release.
I have increased the memory limit to the maximum capacity of my hard drive (which is around 20gb), with no success.
You'd need to increase the 32-bit limit of Windows, too ....
I am running the RGUI on Windows XP with 512 mb of RAM. Would anyone have any suggestions as to how I can overcome this problem? I would be most grateful for any help.
No. The distance matrix is 8Gb all by itself.
Brian D. Ripley, ripley at stats.ox.ac.uk Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UK Fax: +44 1865 272595
Hello, Thank you all for your replies. I have just discovered that now, for some reason, R does not let me increase its memory limit beyond 4095 Mb. Also, I get a different error message when computing the matrix using daisy() as opposed to dist(): Error: Cannot allocate vector of size 135168 Kb Does this mean that there is no way in which I can compute this distance matrix on the machine I described below? Is there any possible solution to this problem? Thanks again to all, Arnav. ----- Original Message ----- From: "Prof Brian Ripley" <ripley at stats.ox.ac.uk> To: "Arnav Sheth" <sheth at economics.rutgers.edu> Cc: "R-Help" <R-help at stat.math.ethz.ch> Sent: Wednesday, February 25, 2004 2:12 PM Subject: Re: [R] Computing very large distance matrix
On Wed, 25 Feb 2004, Arnav Sheth wrote:
Hello All, I have a 131072x132 matrix for which I need to compute a regular
euclidean distance matrix, which I then need to transform and run agnes() on this transformed matrix. I am having trouble computing the distance matrix as it is fairly large and I am sure I have gone over the max.
The specific error I am getting is:
Error in vector("double", length) : negative length vectors are not
allowed
We've improved that message for the next release.
I have increased the memory limit to the maximum capacity of my hard drive (which is around 20gb), with no success.
You'd need to increase the 32-bit limit of Windows, too ....
I am running the RGUI on Windows XP with 512 mb of RAM. Would anyone have any suggestions as to how I can overcome this problem? I would be most grateful for any help.
No. The distance matrix is 8Gb all by itself. -- Brian D. Ripley, ripley at stats.ox.ac.uk Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UK Fax: +44 1865 272595
The problem is not with R; the problem is with Windows, as well as the x86 architecture. You'll never really be able to access more than 3GB of physical memory (for a single process). Accessing more than 4GB requires a 64 bit processor. -roger
Arnav Sheth wrote:
Hello, Thank you all for your replies. I have just discovered that now, for some reason, R does not let me increase its memory limit beyond 4095 Mb. Also, I get a different error message when computing the matrix using daisy() as opposed to dist(): Error: Cannot allocate vector of size 135168 Kb Does this mean that there is no way in which I can compute this distance matrix on the machine I described below? Is there any possible solution to this problem? Thanks again to all, Arnav. ----- Original Message ----- From: "Prof Brian Ripley" <ripley at stats.ox.ac.uk> To: "Arnav Sheth" <sheth at economics.rutgers.edu> Cc: "R-Help" <R-help at stat.math.ethz.ch> Sent: Wednesday, February 25, 2004 2:12 PM Subject: Re: [R] Computing very large distance matrix
On Wed, 25 Feb 2004, Arnav Sheth wrote:
Hello All, I have a 131072x132 matrix for which I need to compute a regular
euclidean distance matrix, which I then need to transform and run agnes() on this transformed matrix. I am having trouble computing the distance matrix as it is fairly large and I am sure I have gone over the max.
The specific error I am getting is:
Error in vector("double", length) : negative length vectors are not
allowed
We've improved that message for the next release.
I have increased the memory limit to the maximum capacity of my hard drive (which is around 20gb), with no success.
You'd need to increase the 32-bit limit of Windows, too ....
I am running the RGUI on Windows XP with 512 mb of RAM. Would anyone have any suggestions as to how I can overcome this problem? I would be most grateful for any help.
No. The distance matrix is 8Gb all by itself. -- Brian D. Ripley, ripley at stats.ox.ac.uk Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UK Fax: +44 1865 272595
______________________________________________ R-help at stat.math.ethz.ch mailing list https://www.stat.math.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Roger D. Peng wrote:
The problem is not with R; the problem is with Windows, as well as the x86 architecture. You'll never really be able to access more than 3GB of physical memory (for a single process). Accessing more than 4GB requires a 64 bit processor. -roger
Alternatively, if you move to Linux you could compile a kernel with support for large amounts memory (>4GB) which doesn't require a 64 bit processor. Joel