Regression evaluation

Are there any utilities/packages for showing various performance
metrics of a regression model on some labeled test data?  Basic stuff
I can easily write like RMSE, R-squared, etc., but maybe with some
extra utilities for visualization, or reporting the distribution of
prediction confidence/variance, or other things I haven't thought of.
This is usually reported in most training utilities (like caret's
train), but only over the training data (AFAICT).  Thanks in advance.