Percentiles with R for a big data.frame

An embedded and charset-unspecified text was scrubbed...
Name: not available
URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20130122/cdc26d67/attachment.pl>
Dear R users,

I came up to a problem dealing with percentiles in R.

From my previous questions: I do have a big data.frame, with lots of
columns and rows. The following command enables me to calculate means for
all data frame.

dat1$newID<-rep(1:(nrow(dat1)/12),each=12) #if nrow(dat1)/12 is integer

dat2<-with(dat1,aggregate(cbind(dat1[,1:71]),by=list(newID),mean))

What I need is to calculate percentiles for each group (there are 12 values
in a group). I tried the following:

duomenai<-with(dat1,aggregate(cbind(dat1[,1:71]),by=list(newID),quantiles,0.1,type=4))
You didn't define quantiles, so that won't work.  Assuming that's a 
typo, and you meant quantile...

First, is the following syntax is right?
Secondly, I tried to calculate percentiles using OpenOffice and there is
disagreement between values. If I do calculation for some number row, than
R and OpenOffice numbers coincide, but for a data.frame it seams that
something goes wrong.
There are lots of different formulas for empirical quantiles.  The ones 
available in R are described in the ?quantile help topic.  What formula 
does OpenOffice use?

Duncan Murdoch
An embedded and charset-unspecified text was scrubbed...
Name: not available
URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20130122/3e9dfb32/attachment.pl>

Hey Duncan,

Neither me do imagine what formula OpenOffice uses for quantiles. I  
have
checked a data string, 24 values, to calculate a quantiles with  
OpenOffice
and R. The result is identical. The problem arises when I try to  
implement
quantile calculation in this form:
dat2<-with(dat1,aggregate(cbind(dat1[, 
1:71]),by=list(newID),quantiles,0.1,type=4))
. This code does not generate an error, but I guess neither a right  
result.
You guess? What result and what is "right"?
So my question would be:
How I could calculate quantiles for a big data.frame in R (71  
columns and
288 rows). I need to take 24 rows, calculate quantiles, then take  
another
24 rows etc..for 71 columns.

You have already been told that you are misspelling the name of the R  
function.

The other open question in my mind is whether you were hoping for  
something other than a single quantile (in this case the 10th  
percentile, or perhaps wanted the quantiles that would divide your  
data into deciles?

If you want to do the calculation within groups then the second  
argument to `aggregate` must specify the grouping. By design  
`aggregate` will apply the function on all columns.
David.

> Thanks in advance.
>
>
>
>
> 2013/1/22 Duncan Murdoch <murdoch.duncan at gmail.com>
>
>> On 13-01-21 6:41 PM, Simonas Kecorius wrote:
>>
>>> Dear R users,
>>>
>>> I came up to a problem dealing with percentiles in R.
>>>
>>> From my previous questions: I do have a big data.frame, with lots of
>>>>
>>> columns and rows. The following command enables me to calculate  
>>> means for
>>> all data frame.
>>>
>>> dat1$newID<-rep(1:(nrow(dat1)/**12),each=12) #if nrow(dat1)/12 is  
>>> integer
>>>
>>> dat2<-with(dat1,aggregate(**cbind(dat1[, 
>>> 1:71]),by=list(**newID),mean))
>>>
>>> What I need is to calculate percentiles for each group (there are 12
>>> values
>>> in a group). I tried the following:
>>>
>>> duomenai<-with(dat1,aggregate(**cbind(dat1[,1:71]),by=list(**
>>> newID),quantiles,0.1,type=4))
>>>
>>
>> You didn't define quantiles, so that won't work.  Assuming that's a  
>> typo,
>> and you meant quantile...
>>
>>
>>>
>>> First, is the following syntax is right?
>>> Secondly, I tried to calculate percentiles using OpenOffice and  
>>> there is
>>> disagreement between values. If I do calculation for some number  
>>> row, than
>>> R and OpenOffice numbers coincide, but for a data.frame it seams  
>>> that
>>> something goes wrong.
>>>
>>
>> There are lots of different formulas for empirical quantiles.  The  
>> ones
>> available in R are described in the ?quantile help topic.  What  
>> formula
>> does OpenOffice use?
>>
>> Duncan Murdoch
>>
>>
>
>
> -- 
> Simonas Kecorius
> **
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

David Winsemius, MD
Alameda, CA, USA
An embedded and charset-unspecified text was scrubbed...
Name: not available
URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20130123/40115da2/attachment.pl>

I found a code:

y.ts <- ts(data, frequency=12)
aggregate(y.ts, FUN=quantile, probs=0.10)

Seems it works fine even for a big data.frame.
Except for the fact that 'y.ts' is not a dataframe, so you are using a  
function that has different arguments than `aggregate.data.frame`.  
With the `ts` call you implicitly constructed  `ts(data.matrix(data),  
frequency=12)` and will be getting quantile estimates on groups of 12,  
which is not at all what you asked for in the first place.
David.
>
> Thanks for your help.
>
> 2013/1/22 David Winsemius <dwinsemius at comcast.net>
>
> On Jan 22, 2013, at 5:58 AM, Simonas Kecorius wrote:
>
> Hey Duncan,
>
> Neither me do imagine what formula OpenOffice uses for quantiles. I  
> have
> checked a data string, 24 values, to calculate a quantiles with  
> OpenOffice
> and R. The result is identical. The problem arises when I try to  
> implement
> quantile calculation in this form:
> dat2<-with(dat1,aggregate(cbind(dat1[, 
> 1:71]),by=list(newID),quantiles,0.1,type=4))
> . This code does not generate an error, but I guess neither a right  
> result.
>
> You guess? What result and what is "right"?
>
>
> So my question would be:
> How I could calculate quantiles for a big data.frame in R (71  
> columns and
> 288 rows). I need to take 24 rows, calculate quantiles, then take  
> another
>
> 24 rows etc..for 71 columns.
>
>
> You have already been told that you are misspelling the name of the  
> R function.
>
> The other open question in my mind is whether you were hoping for  
> something other than a single quantile (in this case the 10th  
> percentile, or perhaps wanted the quantiles that would divide your  
> data into deciles?
>
> If you want to do the calculation within groups then the second  
> argument to `aggregate` must specify the grouping. By design  
> `aggregate` will apply the function on all columns.
> -- 
> David.
>
> Thanks in advance.
>
>
>
>
> 2013/1/22 Duncan Murdoch <murdoch.duncan at gmail.com>
>
> On 13-01-21 6:41 PM, Simonas Kecorius wrote:
>
> Dear R users,
>
> I came up to a problem dealing with percentiles in R.
>
> From my previous questions: I do have a big data.frame, with lots of
>
> columns and rows. The following command enables me to calculate  
> means for
> all data frame.
>
> dat1$newID<-rep(1:(nrow(dat1)/**12),each=12) #if nrow(dat1)/12 is  
> integer
>
> dat2<-with(dat1,aggregate(**cbind(dat1[,1:71]),by=list(**newID),mean))
>
>
> What I need is to calculate percentiles for each group (there are 12
> values
> in a group). I tried the following:
>
> duomenai<-with(dat1,aggregate(**cbind(dat1[,1:71]),by=list(**
> newID),quantiles,0.1,type=4))
>
>
> You didn't define quantiles, so that won't work.  Assuming that's a  
> typo,
> and you meant quantile...
>
>
>
> First, is the following syntax is right?
> Secondly, I tried to calculate percentiles using OpenOffice and  
> there is
> disagreement between values. If I do calculation for some number  
> row, than
> R and OpenOffice numbers coincide, but for a data.frame it seams that
> something goes wrong.
>
>
> There are lots of different formulas for empirical quantiles.  The  
> ones
> available in R are described in the ?quantile help topic.  What  
> formula
> does OpenOffice use?
>
> Duncan Murdoch
>
>
>
>
> -- 
> Simonas Kecorius
> **
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
> David Winsemius, MD
> Alameda, CA, USA
>
>
>
>
> -- 
> Simonas Kecorius
>

David Winsemius, MD
Alameda, CA, USA
An embedded and charset-unspecified text was scrubbed...
Name: not available
URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20130125/6a60cbf5/attachment.pl>