Dear all,
I'm still not sure that this list is the right place for that. So if
those questions bother you, please give me the name of the list I should
contact.
Here a version of the script I'm using to download MODIS data from NSIDC
(weekly snow cover)
Tsnow <-
"ftp://anonymous:test at n4ftl01u.ecs.nasa.gov/SAN/MOST/MOD10A2.005/"
###server of NSIDC with directories / id and password
myfolder<-paste(tempdir(),"\\" , sep="") #the folder to download data
#####-getting the list of the folders in Tsnow (thanks to Barry Rowlingson)
resp <-getURL(Tsnow)
items <- strsplit(resp,"\r\n")[[1]]
folderLines <- items[substr(items,1,1)=='d']
lastBit <- function(x){x[length(x)]}
dates <- unlist(lapply(strsplit(folderLines," "),lastBit))
datesK<-data.frame(dates=dates, keep=rep(NA,length(dates)))
##### select months for which I want the data
months<-c("01","02","03","04","05","06","10", "11", "12") #months for
which I want the data
for (i in months){ #loop to keep only the dates I'm interested in
garde<-grep(datesK$dates, pattern=paste(".",i,".", sep=""))
datesK$keep[garde]<-1
}
datesK<-subset(datesK, datesK$keep==1) # keep the dates of interest
for (i in 1:length(datesK[,1])){ ## loop to download the tiles of
interest
getlist1 <- strsplit(getURL(paste(Tsnow, datesK[i,][[1]], "/",
sep=""),.opts=curlOptions(ftplistonly=TRUE)), "\r\n")[[1]] #get the
files in the folder i
BLOCKtemp <- getlist1[grep(getlist1,
pattern="MOD10A2.********.h18v04.005.*************.hdf")[1]] #select the
tile of interest (in my case h18v04.005)
download.file(paste(Tsnow, datesK[i,][[1]], "/",
BLOCKtemp,sep=""), destfile=paste(myfolder, BLOCKtemp, sep=""),
mode='wb', method='wget') #download the tile
}
So here is my problem. I can't get into some specifc folders and receive
the error message.
Error in curlPerform(curl = curl, .opts = opts, .encoding = .encoding) :
server did not report OK, got 550
For example, i=4 (folder "2000.03.13"). And this is always true,
suggesting it's not a problem of connection. The error appears as soon
as getURL is called in the loop.
I succeed in downloading for some values of i (manually) but when
looping, it stops on those "black" folders. I've checked directly from
filezilla or my web browser: I can get into those folder and download
the tiles.
The tiles are pretty small (150 to 450 Ko). So if someone could try from
his place to see if he gets the same error message for the same value of
i (4, 28,31, etc.).
Since I have 550 tiles to download and process, I would prefer not to do
that manually (and I can't get to use the tool proposed by the NSIDC for
the moment).
Thanks for the help
Alex
P.S.: my next question will deal with some projection problem !!!
Alexandre Villers
PhD Student
Team "Biodiversity"
Centre d'Etudes Biologiques de Chiz?-CNRS UPR1934
79360 Beauvoir sur Niort
Phone +33 (0)5 49 09 96 13
Fax +33 (0)5 49 09 65 26
__________ Information from ESET Mail Security, version of virus signature database 4244 (20090715) __________
The message was checked by ESET Mail Security.
http://www.eset.com
-----Original Message-----
From: r-sig-geo-bounces at stat.math.ethz.ch [mailto:r-sig-geo-bounces at stat.math.ethz.ch] On Behalf
Of Alexandre VILLERS
Sent: Wednesday, July 15, 2009 10:03 AM
To: Aide R SIG GEO
Subject: [R-sig-Geo] still struggling with FTP download on MODIS
Dear all,
I'm still not sure that this list is the right place for that. So if
those questions bother you, please give me the name of the list I should
contact.
Here a version of the script I'm using to download MODIS data from NSIDC
(weekly snow cover)
Tsnow <-
"ftp://anonymous:test at n4ftl01u.ecs.nasa.gov/SAN/MOST/MOD10A2.005/"
###server of NSIDC with directories / id and password
myfolder<-paste(tempdir(),"\\" , sep="") #the folder to download data
#####-getting the list of the folders in Tsnow (thanks to Barry Rowlingson)
resp <-getURL(Tsnow)
items <- strsplit(resp,"\r\n")[[1]]
folderLines <- items[substr(items,1,1)=='d']
lastBit <- function(x){x[length(x)]}
dates <- unlist(lapply(strsplit(folderLines," "),lastBit))
datesK<-data.frame(dates=dates, keep=rep(NA,length(dates)))
##### select months for which I want the data
months<-c("01","02","03","04","05","06","10", "11", "12") #months for
which I want the data
for (i in months){ #loop to keep only the dates I'm interested in
garde<-grep(datesK$dates, pattern=paste(".",i,".", sep=""))
datesK$keep[garde]<-1
}
datesK<-subset(datesK, datesK$keep==1) # keep the dates of interest
for (i in 1:length(datesK[,1])){ ## loop to download the tiles of
interest
getlist1 <- strsplit(getURL(paste(Tsnow, datesK[i,][[1]], "/",
sep=""),.opts=curlOptions(ftplistonly=TRUE)), "\r\n")[[1]] #get the
files in the folder i
BLOCKtemp <- getlist1[grep(getlist1,
pattern="MOD10A2.********.h18v04.005.*************.hdf")[1]] #select the
tile of interest (in my case h18v04.005)
download.file(paste(Tsnow, datesK[i,][[1]], "/",
BLOCKtemp,sep=""), destfile=paste(myfolder, BLOCKtemp, sep=""),
mode='wb', method='wget') #download the tile
}
So here is my problem. I can't get into some specifc folders and receive
the error message.
Error in curlPerform(curl = curl, .opts = opts, .encoding = .encoding) :
server did not report OK, got 550
The ftp error code "550" means: "Requested action not taken. File unavailable (e.g., file not found,
no access).", which means that the ftp server is not allowing you access.
Did you try checking your wget tool? I remember I had headaches before I manage to get the wget
running without problems. Make sure you set:
options(download.file.method="auto")
wget.exe you need to put in your Windows directory folder (Note: make sure you disable your
antivirus tools such as Norton or McAfee otherwise it might block wget from running);
For example, i=4 (folder "2000.03.13"). And this is always true,
suggesting it's not a problem of connection. The error appears as soon
as getURL is called in the loop.
I succeed in downloading for some values of i (manually) but when
looping, it stops on those "black" folders. I've checked directly from
filezilla or my web browser: I can get into those folder and download
the tiles.
The tiles are pretty small (150 to 450 Ko). So if someone could try from
his place to see if he gets the same error message for the same value of
i (4, 28,31, etc.).
Since I have 550 tiles to download and process, I would prefer not to do
that manually (and I can't get to use the tool proposed by the NSIDC for
the moment).
Thanks for the help
Alex
P.S.: my next question will deal with some projection problem !!!
--
Alexandre Villers
PhD Student
Team "Biodiversity"
Centre d'Etudes Biologiques de Chiz?-CNRS UPR1934
79360 Beauvoir sur Niort
Phone +33 (0)5 49 09 96 13
Fax +33 (0)5 49 09 65 26
__________ Information from ESET Mail Security, version of virus signature database 4244
(20090715) __________
The message was checked by ESET Mail Security.
http://www.eset.com