Operationally it is a fairly simple process to "integrate out"
any fixed effect parameters via the Laplace approximation in nonlinear
mixed models using AD Model Builder. This could be used to
provide a general "REML like" estimation procedure. These ideas
are not new, See
Title: Integrated likelihood methods for eliminating nuisance parameters
Author(s): Berger JO, Liseo B, Wolpert RL
Source: STATISTICAL SCIENCE Volume: 14 Issue: 1 Pages: 1-22
Published: FEB 1999
Of course how well a particular approach works in practice could
be investigated via simulation.