aggregate function - na.action

Looking at the timings by each stage may help :

? system.time(dt <- data.table(dat))

? user ?system elapsed
? 1.20 ? ?0.28 ? ?1.48

? system.time(setkey(dt, x1, x2, x3, x4, x5, x6, x7, x8)) ? # sort by the
8 columns (one-off)

? user ?system elapsed
? 4.72 ? ?0.94 ? ?5.67

? system.time(udt <- dt[, list(y = sum(y, na.rm = TRUE)), by = 'x1, x2,
x3, x4, x5, x6, x7, x8'])

? user ?system elapsed
? 2.00 ? ?0.21 ? ?2.20 ? ? # compared to 11.07s

data.table doesn't have a custom data structure, so it can't be that.
data.table's structure is the same as data.frame i.e. a list of vectors.
data.table inherits from data.frame. ?It *is* a data.frame, too.

The reasons it is faster in this example include :
1. Memory is only allocated for the largest group.
2. That memory is re-used for each group.
3. Since the data is ordered contiguously in RAM, the memory is copied over
in bulk for each group using
memcpy in C, which is faster than a for loop in C. Page fetches are
expensive; they are minimised.