using poly() to predict

This method of finding yhat as x %*% b works when I use raw polynomials:

x<-1:8
y<- 1+ 1*x + .5*x^2
fit<-lm(y~poly(x,2,raw=T))
b<-coef(fit)
xfit<-seq(min(x),max(x),length=20)
yfit<-b[1] + poly(xfit,2,raw=T) %*% b[-1]
plot(x,y)
lines(xfit,yfit)

But it doesn't work when I use orthogonal polynomials:

fit<-lm(y~poly(x,2))
b<-coef(fit)
yfit<-b[1] + poly(xfit,2) %*% b[-1]
plot(x,y)
lines(xfit,yfit,col='red')

I have a feeling that the second version needs to incorporate poly() coefs
(alpha and norm2) somehow. If so, please tell me how.

I do know how to use predict() for this. I just want to understand how
poly() works.

Thanks very much for any help
Stan