I don't have any experience with your particular problem, but the thing I
notice is that mahalanobis is that by default you specify a covariance
matrix, and it uses solve to calculate its inverse. If you could supply the
inverse covariance matrix (and specify inverted=TRUE to mahalanobis), that
might save a lot of memory.
If you cannot externally calculate the inverse before bringing it into R,
perhaps if you read only the covariance matrix and inverted it first, before
doing anything else? Or perhaps someone else knows some matrix magic?