Dear All,
The code given bellow is to extract values of one region and write that to
a text file(there are 365 binary files in the directory).
The problem which I am facing is that all my files are binary with size of
360 rows and 720 columns.
I specified that in this line: file2<-matrix(data=file,ncol=720,nrow=360)
but I got an error : Error in mean(file2[X, Y], na.rm = TRUE) : subscript
out of bounds.
and then I rewrote the above line as
:file2<-matrix(data=file,ncol=360,nrow=720.I put ncol=360 and nrows =720
which is not right.But that worked and I didn't get any error.however,the
results were not correct.
Any help please
dir1<- list.files("C:\\Users\\aalyaari\\Desktop\\New folder
(10)\\Climate_Rad_f_GAMMA_%d.img", full.names = TRUE)
listfile<-dir()
for (i in c(1:365)) {
conne <- file(listfile[i], "rb")
file<- readBin(conne, double(), size=4, n=720*360, signed=T)
file2<-matrix(data=file,ncol=720,nrow=360)
extract[i]<-mean(file2[X,Y],na.rm=TRUE)
close(conne)
write.table(extract,"C:\\Users\\aalyaari\\Desktop\\New folder
(10)\\sam.txt")}
--
View this message in context: http://r.789695.n4.nabble.com/R-does-not-recognise-columns-and-rows-as-they-are-supposed-to-be-tp4631217.html
Sent from the R help mailing list archive at Nabble.com.
R does not recognise columns and rows as they are supposed to be
12 messages · David Winsemius, Rui Barradas, Jonsson +1 more
On May 24, 2012, at 11:51 AM, Jonsson wrote:
Dear All,
The code given bellow is to extract values of one region and write
that to
a text file(there are 365 binary files in the directory).
The problem which I am facing is that all my files are binary with
size of
360 rows and 720 columns.
I specified that in this line: file2<-
matrix(data=file,ncol=720,nrow=360)
but I got an error : Error in mean(file2[X, Y], na.rm = TRUE) :
subscript
out of bounds.
and then I rewrote the above line as
:file2<-matrix(data=file,ncol=360,nrow=720.I put ncol=360 and nrows
=720
which is not right.But that worked and I didn't get any
error.however,the
results were not correct.
Any help please
dir1<- list.files("C:\\Users\\aalyaari\\Desktop\\New folder
(10)\\Climate_Rad_f_GAMMA_%d.img", full.names = TRUE)
listfile<-dir()
for (i in c(1:365)) {
conne <- file(listfile[i], "rb")
file<- readBin(conne, double(), size=4, n=720*360, signed=T)
file2<-matrix(data=file,ncol=720,nrow=360)
extract[i]<-mean(file2[X,Y],na.rm=TRUE)
close(conne)
write.table(extract,"C:\\Users\\aalyaari\\Desktop\\New folder
(10)\\sam.txt")}
Please do not cross post to SO and Rhelp. (See Posting Guide.) (But if you do, rhelp generally expects reproducible examples which this is not.)
David Winsemius, MD West Hartford, CT
Hello, There are several things with your code, most of which are not errors. 1. X<-c(364:369) ; Y<-c(82:92 and later c(1:365) Expressions of the form m:n are integer sequences, the c() is not needed. 2. extract<-vector() First you create the vector, then keep extending it throughout the loop. This is much slower. If you know the final length (and type), create it like this: extract <- numeric(365) # or double(365) 3. 'dir1' is not used. 4. Now the readBin instruction: 4.1. 'file' is the name of a function, use, say, 'file1' instead. 4.2. Option 'signed' defaults to TRUE but it's only valid for integer types. 5. And, maybe this is the problem, in R, doubles are 8 bytes, not 4. Get rid of option 'size', like the help page says, "If size is specified and not the natural size of the object, each element of the vector is coerced to an appropriate type before being written or as it is read." You are NOT reading doubles. As for the rest, it seems ok to me. (Keep option 'n' in readBin.) Hope this helps, Rui Barradas Jonsson wrote
Dear All,
The code given bellow is to extract values of one region and write that
to a text file(there are 365 binary files in the directory).
The problem which I am facing is that all my files are binary with size of
360 rows and 720 columns.
I specified that in this line:
file2<-matrix(data=file,ncol=720,nrow=360) but I got an error : Error in
mean(file2[X, Y], na.rm = TRUE) : subscript out of bounds.
and then I rewrote the above line as
:file2<-matrix(data=file,ncol=360,nrow=720.I put ncol=360 and nrows =720
which is not right.But that worked and I didn't get any error.however,the
results were not correct.
Any help please
X<-c(364:369) ; Y<-c(82:92) ##### for sellected region
extract<-vector()
dir1<- list.files("C:\\Users\\aalyaari\\Desktop\\New folder
(10)\\Climate_Rad_f_GAMMA_%d.img", full.names = TRUE)
listfile<-dir()
for (i in c(1:365)) {
conne <- file(listfile[i], "rb")
file<- readBin(conne, double(), size=4, n=720*360, signed=T)
file2<-matrix(data=file,ncol=720,nrow=360)
extract[i]<-mean(file2[X,Y],na.rm=TRUE)
close(conne)
write.table(extract,"C:\\Users\\aalyaari\\Desktop\\New folder
(10)\\sam.txt")}
-- View this message in context: http://r.789695.n4.nabble.com/R-does-not-recognise-columns-and-rows-as-they-are-supposed-to-be-tp4631217p4631227.html Sent from the R help mailing list archive at Nabble.com.
Sorry, forgot the last line in the op code. write.table(extract, ...) is called every time through the loop, not just one time after it. Put it after closing '}'. Rui Barradas Rui Barradas wrote
Hello, There are several things with your code, most of which are not errors. 1. X<-c(364:369) ; Y<-c(82:92 and later c(1:365) Expressions of the form m:n are integer sequences, the c() is not needed. 2. extract<-vector() First you create the vector, then keep extending it throughout the loop. This is much slower. If you know the final length (and type), create it like this: extract <- numeric(365) # or double(365) 3. 'dir1' is not used. 4. Now the readBin instruction: 4.1. 'file' is the name of a function, use, say, 'file1' instead. 4.2. Option 'signed' defaults to TRUE but it's only valid for integer types. 5. And, maybe this is the problem, in R, doubles are 8 bytes, not 4. Get rid of option 'size', like the help page says, "If size is specified and not the natural size of the object, each element of the vector is coerced to an appropriate type before being written or as it is read." You are NOT reading doubles. As for the rest, it seems ok to me. (Keep option 'n' in readBin.) Hope this helps, Rui Barradas Jonsson wrote
Dear All,
The code given bellow is to extract values of one region and write that
to a text file(there are 365 binary files in the directory).
The problem which I am facing is that all my files are binary with size
of 360 rows and 720 columns.
I specified that in this line:
file2<-matrix(data=file,ncol=720,nrow=360) but I got an error : Error in
mean(file2[X, Y], na.rm = TRUE) : subscript out of bounds.
and then I rewrote the above line as
:file2<-matrix(data=file,ncol=360,nrow=720.I put ncol=360 and nrows =720
which is not right.But that worked and I didn't get any error.however,the
results were not correct.
Any help please
X<-c(364:369) ; Y<-c(82:92) ##### for sellected region
extract<-vector()
dir1<- list.files("C:\\Users\\aalyaari\\Desktop\\New folder
(10)\\Climate_Rad_f_GAMMA_%d.img", full.names = TRUE)
listfile<-dir()
for (i in c(1:365)) {
conne <- file(listfile[i], "rb")
file<- readBin(conne, double(), size=4, n=720*360, signed=T)
file2<-matrix(data=file,ncol=720,nrow=360)
extract[i]<-mean(file2[X,Y],na.rm=TRUE)
close(conne)
write.table(extract,"C:\\Users\\aalyaari\\Desktop\\New folder
(10)\\sam.txt")}
-- View this message in context: http://r.789695.n4.nabble.com/R-does-not-recognise-columns-and-rows-as-they-are-supposed-to-be-tp4631217p4631229.html Sent from the R help mailing list archive at Nabble.com.
Yes this helped a lot . I exactly followed what you suggested: X<-(82:92) ; Y<-(364:369) ##### for sellected region
extract <- double(365)
setwd("C:\\Users\\aalyaari\\Desktop\\New folder (10)\\")
listfile<-dir()
for (i in 1:365) {
+ conne <- file(listfile[i], "rb") + file1<- readBin(conne, double(), n=360*720) + file2<-matrix(data=file1,ncol=720,nrow=360) + extract[i]<-mean(file2[X,Y],na.rm=TRUE) + close(conne) + write.table(extract,"C:\\Users\\aalyaari\\Desktop\\New folder (10)\\samregion1.txt")} But I wonder why I got all values like -3.75E+306 -1.30E+54 -1.22E+58 and the right ones should be like: 22.25 22.76 33.25 -- View this message in context: http://r.789695.n4.nabble.com/R-does-not-recognise-columns-and-rows-as-they-are-supposed-to-be-tp4631217p4631241.html Sent from the R help mailing list archive at Nabble.com.
X<-c(364:369) ; Y<-c(82:92) ##### for sellected region
...
file2<-matrix(data=file,ncol=720,nrow=360)
extract[i]<-mean(file2[X,Y],na.rm=TRUE)
Note that 2-dimensional subscripts are given as
mat[ROWS, COLUMNS]
and you are using the reverse order, hence you get the
error message that you are asking for rows 364:369 of
a 360-row matrix.
-----Original Message----- From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On Behalf Of Rui Barradas Sent: Thursday, May 24, 2012 9:48 AM To: r-help at r-project.org Subject: Re: [R] R does not recognise columns and rows as they are supposed to be Hello, There are several things with your code, most of which are not errors. 1. X<-c(364:369) ; Y<-c(82:92 and later c(1:365) Expressions of the form m:n are integer sequences, the c() is not needed. 2. extract<-vector() First you create the vector, then keep extending it throughout the loop. This is much slower. If you know the final length (and type), create it like this: extract <- numeric(365) # or double(365) 3. 'dir1' is not used. 4. Now the readBin instruction: 4.1. 'file' is the name of a function, use, say, 'file1' instead. 4.2. Option 'signed' defaults to TRUE but it's only valid for integer types. 5. And, maybe this is the problem, in R, doubles are 8 bytes, not 4. Get rid of option 'size', like the help page says, "If size is specified and not the natural size of the object, each element of the vector is coerced to an appropriate type before being written or as it is read." You are NOT reading doubles.
The file may well contain single precision numbers and using what="double", size=4 will read them and convert them to doubles so other functions in R can use them. (Likewise, readBin can read files containing 1- or 2-byte signed or unsigned integers and store them as the 4-byte signed integers that the rest of R can deal with.) If the file was created on another sort of machine, then you may have worry about the byte order - try adding endian="big" or endian="little".
As for the rest, it seems ok to me. (Keep option 'n' in readBin.) Hope this helps, Rui Barradas Jonsson wrote
Dear All,
The code given bellow is to extract values of one region and write that
to a text file(there are 365 binary files in the directory).
The problem which I am facing is that all my files are binary with size of
360 rows and 720 columns.
I specified that in this line:
file2<-matrix(data=file,ncol=720,nrow=360) but I got an error : Error in
mean(file2[X, Y], na.rm = TRUE) : subscript out of bounds.
and then I rewrote the above line as
:file2<-matrix(data=file,ncol=360,nrow=720.I put ncol=360 and nrows =720
which is not right.But that worked and I didn't get any error.however,the
results were not correct.
Any help please
X<-c(364:369) ; Y<-c(82:92) ##### for sellected region
extract<-vector()
dir1<- list.files("C:\\Users\\aalyaari\\Desktop\\New folder
(10)\\Climate_Rad_f_GAMMA_%d.img", full.names = TRUE)
listfile<-dir()
for (i in c(1:365)) {
conne <- file(listfile[i], "rb")
file<- readBin(conne, double(), size=4, n=720*360, signed=T)
file2<-matrix(data=file,ncol=720,nrow=360)
extract[i]<-mean(file2[X,Y],na.rm=TRUE)
close(conne)
write.table(extract,"C:\\Users\\aalyaari\\Desktop\\New folder
(10)\\sam.txt")}
-- View this message in context: http://r.789695.n4.nabble.com/R-does-not-recognise- columns-and-rows-as-they-are-supposed-to-be-tp4631217p4631227.html Sent from the R help mailing list archive at Nabble.com.
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Yes I exactly followed what you all suggested: X<-(82:92) ; Y<-(364:369) ##### for sellected region
extract <- double(365)
setwd("C:\\Users\\aalyaari\\Desktop\\New folder (10)\\")
listfile<-dir()
for (i in 1:365) {
+ conne <- file(listfile[i], "rb")
+ file1<- readBin(conne, double(), n=360*720)
+ file2<-matrix(data=file1,ncol=720,nrow=360)
+ extract[i]<-mean(file2[X,Y],na.rm=TRUE)
+ close(conne) }
write.table(extract,"C:\\Users\\aalyaari\\Desktop\\New folder
(10)\\samregion1.txt")
But I am still getting(negative values) all values like:
-3.75E+306
-1.30E+54
-1.22E+58
and the right ones should be like:
22.25
22.76
33.25
--
View this message in context: http://r.789695.n4.nabble.com/R-does-not-recognise-columns-and-rows-as-they-are-supposed-to-be-tp4631217p4631279.html
Sent from the R help mailing list archive at Nabble.com.
Is the file format documented?
If not you can search for a possible format if you know the values
at the start of the file. For one of your files, show the results of the following:
file <- "your filename here"
for(what in c("double", "integer")) {
for(size in c(4, 8)) {
for(endian in c("little", "big")) {
cat(sep="", what, "/", size, "/", endian, ":\n ");
print(readBin(file, what=what, size=size, endian=endian, n=6))
}
}
}
Do any of them look ok? If not you may want to loop over possible
offsets in the file by opening a connection, reading from 1 to 7 one-byte
integers, and then reading the data of interest.
Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com
-----Original Message----- From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On Behalf Of Jonsson Sent: Thursday, May 24, 2012 11:41 PM To: r-help at r-project.org Subject: Re: [R] R does not recognise columns and rows as they are supposed to be Yes I exactly followed what you all suggested: X<-(82:92) ; Y<-(364:369) ##### for sellected region
extract <- double(365)
setwd("C:\\Users\\aalyaari\\Desktop\\New folder (10)\\")
listfile<-dir()
for (i in 1:365) {
+ conne <- file(listfile[i], "rb")
+ file1<- readBin(conne, double(), n=360*720)
+ file2<-matrix(data=file1,ncol=720,nrow=360)
+ extract[i]<-mean(file2[X,Y],na.rm=TRUE)
+ close(conne) }
write.table(extract,"C:\\Users\\aalyaari\\Desktop\\New folder
(10)\\samregion1.txt")
But I am still getting(negative values) all values like:
-3.75E+306
-1.30E+54
-1.22E+58
and the right ones should be like:
22.25
22.76
33.25
--
View this message in context: http://r.789695.n4.nabble.com/R-does-not-recognise-
columns-and-rows-as-they-are-supposed-to-be-tp4631217p4631279.html
Sent from the R help mailing list archive at Nabble.com.
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Yes I did so. Yes the first values are the right ones: -9999 -9999. so this meant that I should consider my data as: double/4/little Is it so? file <- "C:\\Users\\aalyaari\\Documents\\INRA\\WFD_reprocessed\\dialyswco\\2001\\SWdown_200101_01.img"
for(what in c("double", "integer")) {
+ for(size in c(4, 8)) {
+ for(endian in c("little", "big")) {
+ cat(sep="", what, "/", size, "/", endian, ":\n ");
+ print(readBin(file, what=what, size=size, endian=endian, n=6))
+ }
+ }
+ }
double/4/little:
[1] -9999 -9999 -9999 -9999 -9999 -9999
double/4/big:
[1] 5.520452e-39 5.520452e-39 5.520452e-39 5.520452e-39 5.520452e-39
5.520452e-39
double/8/little:
[1] -5.592396e+29 -5.592396e+29 -5.592396e+29 -5.592396e+29 -5.592396e+29
-5.592396e+29
double/8/big:
[1] 1.563804e-307 1.563804e-307 1.563804e-307 1.563804e-307 1.563804e-307
1.563804e-307
integer/4/little:
[1] -971228160 -971228160 -971228160 -971228160 -971228160 -971228160
integer/4/big:
[1] 3939526 3939526 3939526 3939526 3939526 3939526
integer/8/little:
[1] -971228160 -971228160 -971228160 -971228160 -971228160 -971228160
integer/8/big:
[1] 3939526 3939526 3939526 3939526 3939526 3939526
-- View this message in context: http://r.789695.n4.nabble.com/R-does-not-recognise-columns-and-rows-as-they-are-supposed-to-be-tp4631217p4631344.html Sent from the R help mailing list archive at Nabble.com.
In the absence of documentation for the file format, what="double",size=4,endian="little" would be a good guess. Many problems people report on this list are due to errors in reading data into R. Converting data from one format to another is always error-prone and you need to check that the conversion was done correctly. Don't try to analyze the imported data until you have checked the import process. E.g., you think your files should contain 360*720 4-byte values, so look at the number of bytes in the file with file.info(file)$size and see if it matches the expected 360*720*4. After you read the data into the matrix, x, look at the quartiles with quantile(x). Do those -9999's represent missing values? If so, make them NA's with is.na(x) <- x==-9999 and look at the quartiles again, with quantile(x,na.rm=TRUE). Take a look at the pattern of the data with image(x) or image(array(rank(x), dim=dim(x))) (or plot(x[,10]) or plot(x[20,]), etc.). If any these things look odd, fix the import procedure before doing the "real" analysis of the data. Bill Dunlap Spotfire, TIBCO Software wdunlap tibco.com
-----Original Message----- From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On Behalf Of Jonsson Sent: Friday, May 25, 2012 8:58 AM To: r-help at r-project.org Subject: Re: [R] R does not recognise columns and rows as they are supposed to be Yes I did so. Yes the first values are the right ones: -9999 -9999. so this meant that I should consider my data as: double/4/little Is it so? file <- "C:\\Users\\aalyaari\\Documents\\INRA\\WFD_reprocessed\\dialyswco\\2001\\SWdow n_200101_01.img"
for(what in c("double", "integer")) {
+ for(size in c(4, 8)) {
+ for(endian in c("little", "big")) {
+ cat(sep="", what, "/", size, "/", endian, ":\n ");
+ print(readBin(file, what=what, size=size, endian=endian, n=6))
+ }
+ }
+ }
double/4/little:
[1] -9999 -9999 -9999 -9999 -9999 -9999
double/4/big:
[1] 5.520452e-39 5.520452e-39 5.520452e-39 5.520452e-39 5.520452e-39
5.520452e-39
double/8/little:
[1] -5.592396e+29 -5.592396e+29 -5.592396e+29 -5.592396e+29 -5.592396e+29
-5.592396e+29
double/8/big:
[1] 1.563804e-307 1.563804e-307 1.563804e-307 1.563804e-307 1.563804e-307
1.563804e-307
integer/4/little:
[1] -971228160 -971228160 -971228160 -971228160 -971228160 -971228160
integer/4/big:
[1] 3939526 3939526 3939526 3939526 3939526 3939526
integer/8/little:
[1] -971228160 -971228160 -971228160 -971228160 -971228160 -971228160
integer/8/big:
[1] 3939526 3939526 3939526 3939526 3939526 3939526
-- View this message in context: http://r.789695.n4.nabble.com/R-does-not-recognise- columns-and-rows-as-they-are-supposed-to-be-tp4631217p4631344.html Sent from the R help mailing list archive at Nabble.com.
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
The trick may be behind not reading the files properly is that all my 365
files donot have the same name. for example:
files from number 1 to number9 are named for the first month:
SWdown_200101_01.img
SWdown_200101_09.img
files from number 10 to number30 are named:
SWdown_200101_10.img
SWdown_200101_30.img
And so on for the second month:
files from number 1 to number9 are named:
SWdown_200102_01.img
SWdown_200102_09.img
files from number 10 to number30 are named:
SWdown_200102_10.img
SWdown_200102_30.img
and in my code I just set the dierctory: dir1<-
list.files("C:\\Users\\aalyaari\\Desktop\\New folder (11)\\", "*.img",
full.names = TRUE).
assuming that R would read files in order.But I do not really know if this
right or I shall specify the names.
X<-(82:85) ; Y<-(364:367) ##### for sellected region
extract <- double()
dir1<- list.files("C:\\Users\\aalyaari\\Desktop\\New folder (11)\\",
"*.bin", full.names = TRUE)
for (i in 1:365) {
conne <- file(dir1[i], "rb")
file1<- readBin(conne, numeric(),size=4, n=360*720, endian="little")
file2<-matrix(data=file1,ncol=720,nrow=360)
extract[i]<-mean(file2[X,Y],na.rm=TRUE)
close(conne) }
write.table(as.double(extract),"C:\\Users\\aalyaari\\Desktop\\New folder
(10)\\new6.txt")
--
View this message in context: http://r.789695.n4.nabble.com/R-does-not-recognise-columns-and-rows-as-they-are-supposed-to-be-tp4631217p4631352.html
Sent from the R help mailing list archive at Nabble.com.
I tried to read only one file and get some information but this is what I
got:
sam=file("C:\\Users\\2001\\SWdown_200101_01.img", "rb")
file1<- readBin(sam, double(),size=4, n=360*720)
file.info(file1)$size
Error in file.info(file1) : invalid filename argument
dim(file1)
NULL
--
View this message in context: http://r.789695.n4.nabble.com/R-does-not-recognise-columns-and-rows-as-they-are-supposed-to-be-tp4631217p4631366.html
Sent from the R help mailing list archive at Nabble.com.