Simple loop

[...]

ave() can deal that problem:

  > cbind(x, newCol2 = with(x, ave(H, Site, Prof,

FUN=function(y)y-min(y))))
    Site Prof  H newCol2
  1    1    1 24       8
  2    1    1 16       0
  3    1    1 67      51
  4    1    2 23       0
  5    1    2 56      33
  6    1    2 45      22
  7    2    1 67      21
  8    2    1 46       0
  Warning message:
  In min(y) : no non-missing arguments to min; returning Inf
The warning is unfortunate: ave() calls FUN even for when
there is no data for a particular group (Site=2, Prof=2 in this
case).

The warning may be avoided using min(y, Inf) instead of min().

  cbind(x, newCol2 = with(x, ave(H, Site, Prof, FUN=function(y)y-min(y,Inf))))

    Site Prof  H newCol2
  1    1    1 24       8
  2    1    1 16       0
  3    1    1 67      51
  4    1    2 23       0
  5    1    2 56      33
  6    1    2 45      22
  7    2    1 67      21
  8    2    1 46       0

Another approach is to combine Site, Prof to a single column
in any way suitable for the application. For example

  cbind(x, newCol2 = with(x, ave(H, paste(Site, Prof), FUN=function(y)y-min(y))))

Petr Savicky.