a fast way to do my job

HI Bert and Ben,



Yes, running lm.fit using the matrix format is much faster. I read a
couple of online comments why it is faster.



However, the residual values for three tested variables or genes from lm
function and lm.fit function are different, with Pearson correlation of
0.55, 0.89, and 0.99.



I have not found the reason.



Thanks,


Ding



*From:* Bert Gunter <bgunter.4567 at gmail.com>
*Sent:* Friday, August 9, 2024 7:11 PM
*To:* Ben Bolker <bbolker at gmail.com>
*Cc:* Yuan Chun Ding <ycding at coh.org>; r-help at r-project.org
*Subject:* Re: [R] a fast way to do my job



Better idea, Ben! It would work as you might expect it to to produce the
same results as the above: ##first make sure your regressor is a matrix:
pur2 <- matrix(purity2, ncol =1) ## convert the data frame variables into a
matrix dat <-

Better idea, Ben!



It would work as you might expect it to to produce the same results as

the above:



##first make sure your regressor is a matrix:

pur2 <- matrix(purity2, ncol =1)

## convert the data frame variables into a matrix

dat <- as.matrix(gem751be.rpkm[ , 74:35164])

##then

result <- residuals(lm.fit( x= pur2, y = dat))



Cheers,

Bert



On Fri, Aug 9, 2024 at 6:38?PM Ben Bolker <bbolker at gmail.com> wrote:

You can also fit a linear model with a matrix-valued response

variable, which should be even faster (not sure off the top of my head

how to get the residuals and reshape them to the dimensions you want)

On Fri, Aug 9, 2024 at 9:31?PM Bert Gunter <bgunter.4567 at gmail.com> wrote:

See ?lm.fit.

I must be missing something, because:

results <- sapply(74:35164, \(i) residuals(lm.fit(purity2,

gem751be.rpkm[, i] )))

would give you a 751 x 35091 matrix of the residuals from each of the

regressions.

I assume it will be considerably faster than all the overhead you are

carrying in your current code, but of course you'll have to try it and

see. ... Assuming that I have interpreted your request correctly.

Ignore if not.

Cheers,

Bert

On Fri, Aug 9, 2024 at 4:50?PM Yuan Chun Ding via R-help

<r-help at r-project.org> wrote:

Dear R users,

I am running the following code below,  the gem751be.rpkm is a dataframe with dim of 751 samples by 35164 variables,  73 phenotypic variables in the furst to 73rd column and 35091 genomic variables or genes in the 74th to 35164th columns.  What I need to do is to calculate the residuals for each gene using the simple linear regression model of genelist[i] ~ purity2;

The following code is running,  it takes long time, but I have an expensive ThinkStation window computer.

Can you provide a fast way to do it?

Thank you,

Ding

---------------------------------------------------------------------------------

gem751be.rpkm <-merge(gem751be10, as.data.frame(t(rna849.fpkm2)),

+                           by.x="id2",by.y=0)

  row.names(gem751be.rpkm)<-gem751be.rpkm$id3

  colnames(gem751be.rpkm)<-gsub(colnames(gem751be.rpkm),pattern="-",replacement="_")

  genelist <- gem751be.rpkm %>% dplyr::select(74:35164)

  residuals <- NULL

  for (i in 1:length(genelist)) {

+     #i=1

+     formula <- reformulate("purity2", response=names(genelist)[i])

+     model <- lm(formula, data = gem751be.rpkm)

+     resi <- as.data.frame(residuals(model))

+     colnames(resi)[1]<-names(genelist)[i]

+     resi <-as.data.frame(t(resi))

+     residuals <- rbind(residuals, resi)

+   }

----------------------------------------------------------------------

------------------------------------------------------------

-SECURITY/CONFIDENTIALITY WARNING-

This message and any attachments are intended solely for the individual or entity to which they are addressed. This communication may contain information that is privileged, confidential, or exempt from disclosure under applicable law (e.g., personal health information, research data, financial information). Because this e-mail has been sent without encryption, individuals other than the intended recipient may be able to view the information, forward it to others or tamper with the information without the knowledge or consent of the sender. If you are not the intended recipient, or the employee or person responsible for delivering the message to the intended recipient, any dissemination, distribution or copying of the communication is strictly prohibited. If you received the communication in error, please notify the sender immediately by replying to this message and deleting the message and any accompanying files from your system. If, due to the security risks, you do not wish to rec

 eive further communications via e-mail, please reply to this message and inform the sender that you do not wish to receive further e-mail from the sender. (LCP301)

------------------------------------------------------------

        [[alternative HTML version deleted]]

https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-help__;!!Fou38LsQmgU!qcJ4z-vlMNzsa8XCsJUcuPOz8Vt12zsV_XaWpqXsyUYJBTlcNRonFPr7w7Ql3xqcDnZ9ZYC8JX72PW30DQ$ <https://urldefense.com/v3/__https:/stat.ethz.ch/mailman/listinfo/r-help__;!!Fou38LsQmgU!qcJ4z-vlMNzsa8XCsJUcuPOz8Vt12zsV_XaWpqXsyUYJBTlcNRonFPr7w7Ql3xqcDnZ9ZYC8JX72PW30DQ$>

https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-help__;!!Fou38LsQmgU!qcJ4z-vlMNzsa8XCsJUcuPOz8Vt12zsV_XaWpqXsyUYJBTlcNRonFPr7w7Ql3xqcDnZ9ZYC8JX72PW30DQ$ <https://urldefense.com/v3/__https:/stat.ethz.ch/mailman/listinfo/r-help__;!!Fou38LsQmgU!qcJ4z-vlMNzsa8XCsJUcuPOz8Vt12zsV_XaWpqXsyUYJBTlcNRonFPr7w7Ql3xqcDnZ9ZYC8JX72PW30DQ$>