Read data from .csv file as a matrix

I have a csv file that contains weather observation (rows) by days (in
columns). 

I open using: 

temp = read.csv("Weather.csv", sep=",") 

and read: 

             X X1.Jan X2.Jan X3.Jan X4.Jan 
1          Min      2      3      4      1 
2          Max      6     10      8      6 
3 Forecast Min      3      1      1      3 
4 Forecast Max      8      7      4      9 

If I type 

mean(temp[2,2:3]) 

I get 

X1.Jan X2.Jan 
     6     10 

The same command on 

y = matrix(1:21, ncol=7) 

mean(y[2,2:3]) 

[1] 6.5 

Works because the data is in a matrix. I believe R stores the data from the
csv file as a data.frame with these annoying headers. So how do I convert
the data from my csv file into a matrix? 

I tried as.matrix but it did not help.