Ryan, have you got some links for what you talk about ? I'm quite surprised that there are already packages for doing that because a lot of geneticists are using ASREML even if they have to pay for it. I really hope that there is a good reason for that !
Sorry, I missed this the first time. MCMCglmm allows an arbitrary correlation matrix for random effects (ginverse options) and has a built-in for numerator-relatedness-matrix given pedigree. AnimalINLA also has a built-in for numerator-relatedness-matrix given pedigree. If that parameterization is awkward/slow, you can also use the decomposition trick in either. That is, let K be your matrix, and U %*% D %*% t(U) its cholesky factorization. Then you can set the RE design matrix Z = U %*% sqrt(D) with an identity covariance matrix; in MCMCglmm that's idv(Z) and in inla a f( ..., model="z") . Both these packages rely for speed on Z and or COV(RE) or its inverse being sparse, so I sometimes play with using the PMA package to compute a sparse approximate SVD. Presumably an incomplete cholesky factorization could do the same thing. ASREML is probably worth the (NIH's) money; my understanding is that it's fast, flexible, and robust. I don't know if the above have been designed with very large datasets in mind. Ryan King