Slow computation in for loop

I suspect that your problem comes from the rbind(). I have also noticed an
exponentially slower execution with the increase of the size of the data
frame that you rbind()s. It is much faster to rbind() several separated
temporary data frames (let's say, ten by ten loops), and then to rbind()
them all together.

An even better solution is to allocate a data frame or matrix with the final
size (it seems you can predict it in your example), and then use
df[n, ] <- result
where df is your allocated data frame and n is the iteration.

Best,

Philippe Grosjean

...........]<(({?<...............<?}))><...............................
 ) ) ) ) )
( ( ( ( (       Dr. Philippe Grosjean
 ) ) ) ) )
( ( ( ( (       LOV, UMR 7093
 ) ) ) ) )      Station Zoologique
( ( ( ( (       Observatoire Oceanologique
 ) ) ) ) )      BP 28
( ( ( ( (       06234 Villefranche sur mer cedex
 ) ) ) ) )      France
( ( ( ( (
 ) ) ) ) )      tel: +33.4.93.76.38.18, fax: +33.4.93.76.38.34
( ( ( ( (
 ) ) ) ) )      e-mail: phgrosjean at sciviews.org
( ( ( ( (       SciViews project coordinator (http://www.sciviews.org)
 ) ) ) ) )
.......................................................................



-----Original Message-----
From: r-help-bounces at stat.math.ethz.ch
[mailto:r-help-bounces at stat.math.ethz.ch]On Behalf Of Yves Brostaux
Sent: mercredi 28 mai 2003 12:03
To: r-help at stat.math.ethz.ch
Cc: Peter Dalgaard BSA; Patrick Burns
Subject: Re: [R] Slow computation in for loop


First of all, thank you for your response.

I actually have to refine my pseudocode. 'result' is a numerical vector of
length 7, and is binded with whole results through an rbind() :

for (k in replicates) {
   data <- sampling from a population
   for (i in param1) {
     for (j in param2) {
        result <- function(i, j, data)
        all.results <- rbind(all.results, result)
     }
   }
}

all.result is at most a 220 rows and 7 columns data frame, which doesn't
seem to be big enough to explain such a slow computation.

Moreover, previous computations with a sample size of 100, which took
individually about 4 seconds at most, ran effectively in a little bit more
than 15 minutes for the whole set.

The problem arise with a sample size of 500, increasing single function
computation time normally, but not the whole process !?

Yves Brostaux <brostaux.y at fsagx.ac.be> writes:

Dear members,

I'm using R to do some test computation on a set of parameters of a
function. This function is included in three for() loops, first one
for replications, and the remaining two cycling through possible
parameters values, like this :

for (k in replicates) {
   data <- sampling from a population
   for (i in param1) {
     for (j in param2) {
        result <- function(i, j, data)
     }
   }
}

With the 'hardest' set of parameters, a single computation of the
function take about 16s on an old Sun Sparc workstation with 64 Mb RAM
and don't access a single time to disk.

But when I launch the for() loops (which generate 220 function calls),
disk gets very sollicitated and the whole process takes as much as 8
to 10 hours, instead of the expected 1 hour.

What's wrong here ? Is there a thing I don't know about for() loops,
and a way to correct it ?

The problem with pseudocode: You didn't really overwrite the "result"
every time did you? I bet you stored it somewhere.

Two common causes of inefficiency are (a) that the stored objects may
be large and (b) some naive ways of storing the results involve
copying all preceding results, e.g.

list.of.results <- list()
for (.....){
     result <- ...
     list.of.results <- c(list.of.results, result)
}

The fix for (a) is to extract what you need and discard the rest
and for (b) to allocate the list up front with the proper length and
assign to list.of.results[[i]].

--
   O__  ---- Peter Dalgaard             Blegdamsvej 3
  c/ /'_ --- Dept. of Biostatistics     2200 Cph. N
 (*) \(*) -- University of Copenhagen   Denmark      Ph: (+45) 35327918
~~~~~~~~~~ - (p.dalgaard at biostat.ku.dk)             FAX: (+45) 35327907

______________________________________________
R-help at stat.math.ethz.ch mailing list
https://www.stat.math.ethz.ch/mailman/listinfo/r-help