Hi, I have a 5x731 array A, and I want to compute the sums of the columns. Currently I do: apply(A, 2, sum) But it turns out, this is slow: 70% of my CPU time is spent here, even though there are many complicated steps in my computation. Is there a faster way? Thanks, Martin
Computing sums of the columns of an array
8 messages · Martin C. Martin, Duncan Murdoch, Uwe Ligges +1 more
On 8/5/2005 12:16 PM, Martin C. Martin wrote:
Hi, I have a 5x731 array A, and I want to compute the sums of the columns. Currently I do: apply(A, 2, sum) But it turns out, this is slow: 70% of my CPU time is spent here, even though there are many complicated steps in my computation. Is there a faster way?
You'd probably do better with matrix multiplication: rep(1, nrow(A)) %*% A Duncan Murdoch
Martin C. Martin wrote:
Hi, I have a 5x731 array A, and I want to compute the sums of the columns. Currently I do: apply(A, 2, sum)
colSums(A) Uwe Ligges
But it turns out, this is slow: 70% of my CPU time is spent here, even though there are many complicated steps in my computation. Is there a faster way? Thanks, Martin
______________________________________________ R-help at stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
On Fri, 2005-08-05 at 12:16 -0400, Martin C. Martin wrote:
Hi, I have a 5x731 array A, and I want to compute the sums of the columns. Currently I do: apply(A, 2, sum) But it turns out, this is slow: 70% of my CPU time is spent here, even though there are many complicated steps in my computation. Is there a faster way?
Yes, colSums() e.g.:
set.seed(1234) dat <- matrix(runif(5*731), ncol = 731) system.time(for(i in 1:1000) apply(dat, 2, sum), gcFirst = TRUE)
[1] 8.05 0.00 9.89 0.00 0.00
system.time(for(i in 1:1000) colSums(dat), gcFirst = TRUE)
[1] 0.09 0.01 0.09 0.00 0.00 But neither is that slow on my system. What is A? HTH Gav
%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~% Gavin Simpson [T] +44 (0)20 7679 5522 ENSIS Research Fellow [F] +44 (0)20 7679 7565 ENSIS Ltd. & ECRC [E] gavin.simpsonATNOSPAMucl.ac.uk UCL Department of Geography [W] http://www.ucl.ac.uk/~ucfagls/cv/ 26 Bedford Way [W] http://www.ucl.ac.uk/~ucfagls/ London. WC1H 0AP. %~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%
Duncan Murdoch wrote:
On 8/5/2005 12:16 PM, Martin C. Martin wrote:
Hi, I have a 5x731 array A, and I want to compute the sums of the columns. Currently I do: apply(A, 2, sum) But it turns out, this is slow: 70% of my CPU time is spent here, even though there are many complicated steps in my computation. Is there a faster way?
You'd probably do better with matrix multiplication: rep(1, nrow(A)) %*% A
No, better use colSums(), which has been optimized for this purpose: A <- matrix(seq(1, 10000000), ncol=10000) system.time(colSums(A)) # ~ 0.1 sec. system.time(rep(1, nrow(A)) %*% A) # ~ 0.5 sec. Uwe Ligges
Duncan Murdoch
______________________________________________ R-help at stat.math.ethz.ch mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
On 8/5/2005 12:43 PM, Uwe Ligges wrote:
Duncan Murdoch wrote:
On 8/5/2005 12:16 PM, Martin C. Martin wrote:
Hi, I have a 5x731 array A, and I want to compute the sums of the columns. Currently I do: apply(A, 2, sum) But it turns out, this is slow: 70% of my CPU time is spent here, even though there are many complicated steps in my computation. Is there a faster way?
You'd probably do better with matrix multiplication: rep(1, nrow(A)) %*% A
No, better use colSums(), which has been optimized for this purpose: A <- matrix(seq(1, 10000000), ncol=10000) system.time(colSums(A)) # ~ 0.1 sec. system.time(rep(1, nrow(A)) %*% A) # ~ 0.5 sec.
I didn't claim my solution was the best, only better. :-) One point of interest: I think your example exaggerates the difference by using a matrix of integers. On my machine I get a ratio something like yours with the same example > A <- matrix(seq(1, 10000000), ncol=10000) > system.time(colSums(A)) [1] 0.08 0.00 0.08 NA NA > system.time(rep(1, nrow(A)) %*% A) [1] 0.25 0.01 0.23 NA NA but if I make A floating point, there's much less difference: > A <- matrix(as.numeric(seq(1, 10000000)), ncol=10000) > system.time(colSums(A)) [1] 0.09 0.00 0.09 NA NA > system.time(rep(1, nrow(A)) %*% A) [1] 0.11 0.00 0.12 NA NA Still, colSums is the winner in both cases. Duncan Murdoch
Thanks everyone.
Gavin Simpson wrote:
But neither is that slow on my system. What is A?
It's in the middle of a loop. I'm doing some maximum likelihood estimation, and having to evaluate my model over and over again for different parameter values. Thanks, Martin
Duncan Murdoch wrote:
On 8/5/2005 12:43 PM, Uwe Ligges wrote:
Duncan Murdoch wrote:
On 8/5/2005 12:16 PM, Martin C. Martin wrote:
Hi, I have a 5x731 array A, and I want to compute the sums of the columns. Currently I do: apply(A, 2, sum) But it turns out, this is slow: 70% of my CPU time is spent here, even though there are many complicated steps in my computation. Is there a faster way?
You'd probably do better with matrix multiplication: rep(1, nrow(A)) %*% A
No, better use colSums(), which has been optimized for this purpose: A <- matrix(seq(1, 10000000), ncol=10000) system.time(colSums(A)) # ~ 0.1 sec. system.time(rep(1, nrow(A)) %*% A) # ~ 0.5 sec.
I didn't claim my solution was the best, only better. :-)
>
One point of interest: I think your example exaggerates the difference by using a matrix of integers. On my machine I get a ratio something like yours with the same example
> A <- matrix(seq(1, 10000000), ncol=10000) > system.time(colSums(A))
[1] 0.08 0.00 0.08 NA NA
> system.time(rep(1, nrow(A)) %*% A)
[1] 0.25 0.01 0.23 NA NA but if I make A floating point, there's much less difference:
> A <- matrix(as.numeric(seq(1, 10000000)), ncol=10000) > system.time(colSums(A))
[1] 0.09 0.00 0.09 NA NA
On my machine: [1] 0.12 0.00 0.12 NA NA
> system.time(rep(1, nrow(A)) %*% A)
[1] 0.11 0.00 0.12 NA NA
On my machine: [1] 0.32 0.00 0.32 NA NA Hence still a bigger factor both with R-2.1.1 release (standard BLAS; gcc-3.4.2, WinNT4.0, Athlon XP with real freq. of 1667MHz, 1Gb). And still a bigger factor (0.09 vs. 0.21) on a Xeon 3.06Ghz with 2Gb. Are you using Goto's BLAS (for me, your performance is still not achievable with ATLAS)? Best, Uwe
Still, colSums is the winner in both cases. Duncan Murdoch