I'm writing a program that will tell me whether I should wear a coat, so I'd like to be able to download daily weather forecasts and daily reports of recent past weather conditions. The NOAA has very promising tabular forecasts (http://forecast.weather.gov/MapClick.php?CityName=Ithaca&state=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0&FcstType=digital), but I can't figure out how to import them. Someone must have needed to do this before. Suggestions? Thomas Levine!
Download daily weather data
13 messages · Thomas Levine, Scillieri, John, James Muller +5 more
Thomas, Have a look at the source code for the webpage (ctrl-u in firefox, don't know in internet explorer, etc.). That is what you'd have to parse in order to get the forecast from this page. Typically when I parse webpages such as this I use regular expressions to do so (and I would never downplay the usefulness of regular expressions, but they take a little getting used to). There are two parts to the task: find patterns that allow you to pull out the datum/data you're after; and then write a program to pull it/them out. Also, of course, download the webpage (but that's no issue). I bet you'd be able to find a comma separated value (CSV) file containing the weather report somewhere, which would probably involve a little less labor in order to produce your automatic wardrobe advice. James
On Thu, Feb 26, 2009 at 3:47 PM, Thomas Levine <thomas.levine at gmail.com> wrote:
I'm writing a program that will tell me whether I should wear a coat, so I'd like to be able to download daily weather forecasts and daily reports of recent past weather conditions. The NOAA has very promising tabular forecasts (http://forecast.weather.gov/MapClick.php?CityName=Ithaca&state=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0&FcstType=digital), but I can't figure out how to import them. Someone must have needed to do this before. Suggestions? Thomas Levine!
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Looks like you can sign up to get XML feed data from Weather.com http://www.weather.com/services/xmloap.html Hope it works out! -----Original Message----- From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On Behalf Of James Muller Sent: Thursday, February 26, 2009 3:57 PM To: r-help at r-project.org Subject: Re: [R] Download daily weather data Thomas, Have a look at the source code for the webpage (ctrl-u in firefox, don't know in internet explorer, etc.). That is what you'd have to parse in order to get the forecast from this page. Typically when I parse webpages such as this I use regular expressions to do so (and I would never downplay the usefulness of regular expressions, but they take a little getting used to). There are two parts to the task: find patterns that allow you to pull out the datum/data you're after; and then write a program to pull it/them out. Also, of course, download the webpage (but that's no issue). I bet you'd be able to find a comma separated value (CSV) file containing the weather report somewhere, which would probably involve a little less labor in order to produce your automatic wardrobe advice. James
On Thu, Feb 26, 2009 at 3:47 PM, Thomas Levine <thomas.levine at gmail.com> wrote:
I'm writing a program that will tell me whether I should wear a coat, so I'd like to be able to download daily weather forecasts and daily reports of recent past weather conditions. The NOAA has very promising tabular forecasts (http://forecast.weather.gov/MapClick.php?CityName=Ithaca&state=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0&FcstType=digital), but I can't figure out how to import them. Someone must have needed to do this before. Suggestions? Thomas Levine!
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
This e-mail and any attachments are confidential, may contain legal, professional or other privileged information, and are intended solely for the addressee. If you are not the intended recipient, do not use the information in this e-mail in any way, delete this e-mail and notify the sender. CEG-IP1
Scillieri, John wrote:
Looks like you can sign up to get XML feed data from Weather.com http://www.weather.com/services/xmloap.html
... and use the excellent R package XML by Duncan Temple Lang to parse the document and easily access the data with, e.g.., XPath rather than regular expressions. vQ
Hope it works out! -----Original Message----- From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On Behalf Of James Muller Sent: Thursday, February 26, 2009 3:57 PM To: r-help at r-project.org Subject: Re: [R] Download daily weather data Thomas, Have a look at the source code for the webpage (ctrl-u in firefox, don't know in internet explorer, etc.). That is what you'd have to parse in order to get the forecast from this page. Typically when I parse webpages such as this I use regular expressions to do so (and I would never downplay the usefulness of regular expressions, but they take a little getting used to). There are two parts to the task: find patterns that allow you to pull out the datum/data you're after; and then write a program to pull it/them out. Also, of course, download the webpage (but that's no issue). I bet you'd be able to find a comma separated value (CSV) file containing the weather report somewhere, which would probably involve a little less labor in order to produce your automatic wardrobe advice. James On Thu, Feb 26, 2009 at 3:47 PM, Thomas Levine <thomas.levine at gmail.com> wrote:
I'm writing a program that will tell me whether I should wear a coat, so I'd like to be able to download daily weather forecasts and daily reports of recent past weather conditions. The NOAA has very promising tabular forecasts (http://forecast.weather.gov/MapClick.php?CityName=Ithaca&state=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0&FcstType=digital), but I can't figure out how to import them. Someone must have needed to do this before. Suggestions? Thomas Levine!
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. This e-mail and any attachments are confidential, may contain legal, professional or other privileged information, and are intended solely for the addressee. If you are not the intended recipient, do not use the information in this e-mail in any way, delete this e-mail and notify the sender. CEG-IP1 ______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Yes, as a general thing go to regular expressions if you don't have an existing library available to do the same thing (or you're lazy like me:). Jame On Thu, Feb 26, 2009 at 5:16 PM, Wacek Kusnierczyk
<Waclaw.Marcin.Kusnierczyk at idi.ntnu.no> wrote:
Scillieri, John wrote:
Looks like you can sign up to get XML feed data from Weather.com http://www.weather.com/services/xmloap.html
... and use the excellent R package XML by Duncan Temple Lang to parse the document and easily access the data with, e.g.., XPath rather than regular expressions. vQ
James Muller wrote:
Yes, as a general thing go to regular expressions if you don't have an existing library available to do the same thing (or you're lazy like me:).
many things are simply *much* easier with xpath than with regexes, and with the XML package you got it for free. vQ
Jame On Thu, Feb 26, 2009 at 5:16 PM, Wacek Kusnierczyk <Waclaw.Marcin.Kusnierczyk at idi.ntnu.no> wrote:
Scillieri, John wrote:
Looks like you can sign up to get XML feed data from Weather.com http://www.weather.com/services/xmloap.html
... and use the excellent R package XML by Duncan Temple Lang to parse
the document and easily access the data with, e.g.., XPath rather than
regular expressions.
vQ
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
2009/2/26 Thomas Levine <thomas.levine at gmail.com>:
I'm writing a program that will tell me whether I should wear a coat, so I'd like to be able to download daily weather forecasts and daily reports of recent past weather conditions. The NOAA has very promising tabular forecasts (http://forecast.weather.gov/MapClick.php?CityName=Ithaca&state=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0&FcstType=digital), but I can't figure out how to import them. Someone must have needed to do this before. Suggestions?
You could use my geonames package that uses the GeoNames query service. There's a sample queries here: http://geonames.r-forge.r-project.org/ Easiest is probably to use GNfindNearByWeather: > as.data.frame(GNfindNearByWeather(57,-2)) clouds weatherCondition 1 broken clouds n/a observation windDirection ICAO 1 EGPD 262120Z 25003KT 9000 -RA BKN018 06/05 Q1012 NOSIG 250 EGPD elevation countryCode lng temperature dewPoint windSpeed humidity 1 65 GB -2.216667 6 5 03 93 stationName datetime lat hectoPascAltimeter 1 Aberdeen / Dyce 2009-02-26 21:20:00 57.2 1012 The package is on CRAN. There is of course an easier way to decide if you need to wear a coat, and that is to look out the window :) Barry
See also http://umbrellatoday.com/ Hadley
On Thu, Feb 26, 2009 at 2:47 PM, Thomas Levine <thomas.levine at gmail.com> wrote:
I'm writing a program that will tell me whether I should wear a coat, so I'd like to be able to download daily weather forecasts and daily reports of recent past weather conditions. The NOAA has very promising tabular forecasts (http://forecast.weather.gov/MapClick.php?CityName=Ithaca&state=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0&FcstType=digital), but I can't figure out how to import them. Someone must have needed to do this before. Suggestions? Thomas Levine!
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Perhaps "coat" and "jacket" are more ambiguous in the United States than the United Kingdom. If it's cold enough to warrant it, I wear a jacket in the morning. If it isn't, I don't want to have to carry it around all day. Checking the daily weather forecast is too much work, so I just go by the current temperature in the morning, which leads to many false positives. Thomas Levine!
On Thu, 2009-02-26 at 22:51 +0000, Barry Rowlingson wrote:
2009/2/26 Thomas Levine <thomas.levine at gmail.com>:
I'm writing a program that will tell me whether I should wear a coat, so I'd like to be able to download daily weather forecasts and daily reports of recent past weather conditions. The NOAA has very promising tabular forecasts (http://forecast.weather.gov/MapClick.php?CityName=Ithaca&state=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0&FcstType=digital), but I can't figure out how to import them. Someone must have needed to do this before. Suggestions?
You could use my geonames package that uses the GeoNames query service. There's a sample queries here: http://geonames.r-forge.r-project.org/ Easiest is probably to use GNfindNearByWeather:
> as.data.frame(GNfindNearByWeather(57,-2))
clouds weatherCondition
1 broken clouds n/a
observation windDirection ICAO
1 EGPD 262120Z 25003KT 9000 -RA BKN018 06/05 Q1012 NOSIG 250 EGPD
elevation countryCode lng temperature dewPoint windSpeed humidity
1 65 GB -2.216667 6 5 03 93
stationName datetime lat hectoPascAltimeter
1 Aberdeen / Dyce 2009-02-26 21:20:00 57.2 1012
The package is on CRAN.
There is of course an easier way to decide if you need to wear a
coat, and that is to look out the window :)
Barry
-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: <https://stat.ethz.ch/pipermail/r-help/attachments/20090226/471e9449/attachment-0002.bin>
Dear Thomas, more for the sake of completeness and as an alternative to R. There are GRIB data [1] sets available (some for free) and there is the GPL software Grads [2]. Because the Grib-Format is well documented it should be possible to get it into R easily and make up your own plots/weather analyis. I do not know and have not checked if somebody has already done so. I use this information/tools aside of others during longer-dated off-shore sailing. Best, Bernhard [1] http://www.grib.us/ [2] http://www.iges.org/grads/
-----Urspr?ngliche Nachricht----- Von: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] Im Auftrag von Scillieri, John Gesendet: Donnerstag, 26. Februar 2009 22:58 An: 'James Muller'; 'r-help at r-project.org' Betreff: Re: [R] Download daily weather data Looks like you can sign up to get XML feed data from Weather.com http://www.weather.com/services/xmloap.html Hope it works out! -----Original Message----- From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On Behalf Of James Muller Sent: Thursday, February 26, 2009 3:57 PM To: r-help at r-project.org Subject: Re: [R] Download daily weather data Thomas, Have a look at the source code for the webpage (ctrl-u in firefox, don't know in internet explorer, etc.). That is what you'd have to parse in order to get the forecast from this page. Typically when I parse webpages such as this I use regular expressions to do so (and I would never downplay the usefulness of regular expressions, but they take a little getting used to). There are two parts to the task: find patterns that allow you to pull out the datum/data you're after; and then write a program to pull it/them out. Also, of course, download the webpage (but that's no issue). I bet you'd be able to find a comma separated value (CSV) file containing the weather report somewhere, which would probably involve a little less labor in order to produce your automatic wardrobe advice. James On Thu, Feb 26, 2009 at 3:47 PM, Thomas Levine <thomas.levine at gmail.com> wrote:
I'm writing a program that will tell me whether I should wear a coat, so I'd like to be able to download daily weather forecasts and daily reports of recent past weather conditions. The NOAA has very promising tabular forecasts
=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0>&FcstType=digital),
but I can't figure out how to import them. Someone must have needed to do this before. Suggestions? Thomas Levine!
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. ______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. This e-mail and any attachments are confidential, may contain legal, professional or other privileged information, and are intended solely for the addressee. If you are not the intended recipient, do not use the information in this e-mail in any way, delete this e-mail and notify the sender. CEG-IP1 ______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
*****************************************************************
Confidentiality Note: The information contained in this ...{{dropped:10}}
Geonames unfortunately doesn't have weather forecasts. This is a problem. GRIB looks better. There is an interface between GRIB and R. On Fri, Feb 27, 2009 at 4:14 AM, Pfaff, Bernhard Dr.
<Bernhard_Pfaff at fra.invesco.com> wrote:
Dear Thomas, more for the sake of completeness and as an alternative to R. There are GRIB data [1] sets available (some for free) and there is the GPL software Grads [2]. Because the Grib-Format is well documented it should be possible to get it into R easily and make up your own plots/weather analyis. I do not know and have not checked if somebody has already done so. I use this information/tools aside of others during longer-dated off-shore sailing. Best, Bernhard [1] http://www.grib.us/ [2] http://www.iges.org/grads/
-----Urspr?ngliche Nachricht----- Von: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] Im Auftrag von Scillieri, John Gesendet: Donnerstag, 26. Februar 2009 22:58 An: 'James Muller'; 'r-help at r-project.org' Betreff: Re: [R] Download daily weather data Looks like you can sign up to get XML feed data from Weather.com http://www.weather.com/services/xmloap.html Hope it works out! -----Original Message----- From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On Behalf Of James Muller Sent: Thursday, February 26, 2009 3:57 PM To: r-help at r-project.org Subject: Re: [R] Download daily weather data Thomas, Have a look at the source code for the webpage (ctrl-u in firefox, don't know in internet explorer, etc.). That is what you'd have to parse in order to get the forecast from this page. Typically when I parse webpages such as this I use regular expressions to do so (and I would never downplay the usefulness of regular expressions, but they take a little getting used to). There are two parts to the task: find patterns that allow you to pull out the datum/data you're after; and then write a program to pull it/them out. Also, of course, download the webpage (but that's no issue). I bet you'd be able to find a comma separated value (CSV) file containing the weather report somewhere, which would probably involve a little less labor in order to produce your automatic wardrobe advice. James On Thu, Feb 26, 2009 at 3:47 PM, Thomas Levine <thomas.levine at gmail.com> wrote:
I'm writing a program that will tell me whether I should wear a coat, so I'd like to be able to download daily weather forecasts and daily reports of recent past weather conditions. The NOAA has very promising tabular forecasts
=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0>&FcstType=digital),
but I can't figure out how to import them. Someone must have needed to do this before. Suggestions? Thomas Levine!
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. ______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. This e-mail and any attachments are confidential, may contain legal, professional or other privileged information, and are intended solely for the addressee. ?If you are not the intended recipient, do not use the information in this e-mail in any way, delete this e-mail and notify the sender. CEG-IP1 ______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
*****************************************************************
Confidentiality Note: The information contained in this ...{{dropped:10}}
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Can I just say, it's great to see the R community really come out in support of such a noble and worthy cause as this :). Downfall of civilization, all that. Not here, no! James
On Thu, Feb 26, 2009 at 3:47 PM, Thomas Levine <thomas.levine at gmail.com> wrote:
I'm writing a program that will tell me whether I should wear a coat, so I'd like to be able to download daily weather forecasts and daily reports of recent past weather conditions. The NOAA has very promising tabular forecasts (http://forecast.weather.gov/MapClick.php?CityName=Ithaca&state=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0&FcstType=digital), but I can't figure out how to import them. Someone must have needed to do this before. Suggestions? Thomas Levine!
______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
tlevine wrote:
The NOAA has very promising tabular forecasts (http://forecast.weather.gov/MapClick.php?CityName=Ithaca&state=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0&FcstType=digital), but I can't figure out how to import them.
Sometimes you can just use gsub to get html into R. Try this.
## read html
wf<-readLines("http://forecast.weather.gov/MapClick.php?CityName=Ithaca&state=NY&site=BGM&textField1=42.4422&textField2=-76.5002&e=0&FcstType=digital",
warn=FALSE)
## you do need to find the rows with data (rows 18 and 19), grep may help
here
## mark end of lines
x<-gsub("</tr>", "\n", wf[18:19])
## remove white space...
x<-gsub(" ", "_", x)
## remove degree symbol or add comment.char="" to read.table below
x<-gsub("°", "", x)
## remove html tags
x<-gsub("<[^>]*>", " ", x)
y<-read.table(con<-textConnection(x), fill=TRUE)
close(con)
#now just reshape this mess
# join rows 1,16 and 17-32 (skip 1st two rows with day and hour)
z<-data.frame(rbind( t(y[3:16,-1]), t(y[19:32,-1]) ), row.names=NULL,
stringsAsFactors=FALSE)
names(z)<-substr(y[3:16,1], 1,4)
head(z)
Temp Dewp Wind Wind.1 Wind.2 Gust Sky_ Pcpn Rel. Thun Rain Snow Free Slee
1 49 43 43 18 S 99 40 80 -- Chc -- -- --
2 50 43 44 18 SSW 99 40 77 -- Chc -- -- --
3 50 43 44 17 SW 99 40 77 -- Chc -- -- --
4 48 43 42 16 SW 99 40 83 -- Chc -- -- --
5 45 43 38 15 WSW 100 40 93 -- Chc -- -- --
6 42 42 35 14 W 100 35 100 -- Chc -- -- --
# plot
plot(z["Temp"])
Chris Stubben
View this message in context: http://www.nabble.com/Download-daily-weather-data-tp22233373p22252517.html Sent from the R help mailing list archive at Nabble.com.