Log-likelihood calculation in lme

On a real data set, running the lme function, I get
parameters estimation and a log-likelihood value.
Nevertheless, the variance-covariance matrix in this
case had a determinant close to zero. So, I could not
calculate the log-likelihood myself with the classical
expression.
What is the calculus made in lme?

The evaluation of the log-likelihood used in lme is documented in
chapter 2 of Pinheiro and Bates (Springer, 2000).  The calculation used
in lmer from the lme4 package is somewhat different.  If you wish I can
send you off-list copies of slides from a presentation that explains
that calculation.

I'm not sure which variance-covariance matrix you are referring to but
it is the case that the ML or REML estimates of the variance-covariance
matrix of the random effects can be singular, a fact that is often
ignored in the analysis of data with such models.