OK, finally have something that I like. At least for now:
If you’d like to see the actual page, you can find it linked here -> http://www.daviddemartini.com/WX/
In the next day or so, other changes will be made, so I’m saving this for posterity. I often find it interesting to look back at the genesis of a software project.
The creation of this page is documented here, in a 4 part (4th part still underway) set of posts.
It would be short work to grab the current observations from the local station’s file, then grab the full projected forecast from the Zone’s XML formatted data file (example for Bremerton). I checked the full forecast XML, and unfortunately it does not include the local current conditions. In the grand scheme, this is not a big deal since I’ll need to grab three forecast files already. One more file for current observations is not that much more of a burden.
This section contains the Maximum Forecast Temperatures, covering outlook:
<temperature type="minimum" units="Fahrenheit" time-layout="k-p24h-n7-1"> <name>Daily Minimum Temperature</name> <value>40</value> <value>45</value> <value>43</value> <value>44</value> <value>43</value> <value>43</value> <value>42</value> </temperature>
This section contains the Minimum Forecast Temperatures, covering outlook:
<temperature type="maximum" units="Fahrenheit" time-layout="k-p24h-n7-2"> <name>Daily Maximum Temperature</name> <value>59</value> <value>61</value> <value>60</value> <value>61</value> <value>61</value> <value>60</value> <value>60</value> </temperature>
Also in the file is the projected percentage of precipitation:
<probability-of-precipitation type="12 hour" units="percent" time-layout="k-p12h-n14-1"> <name>12 Hourly Probability of Precipitation</name> <value xsi:nil="true"/> <value xsi:nil="true"/> <value xsi:nil="true"/> <value xsi:nil="true"/> <value xsi:nil="true"/> <value>30</value> <value>30</value> <value xsi:nil="true"/> <value xsi:nil="true"/> <value xsi:nil="true"/> <value xsi:nil="true"/> <value xsi:nil="true"/> <value xsi:nil="true"/> <value xsi:nil="true"/> </probability-of-precipitation>
The XML also is kind enough to include the image names. Based on this, the actual visually rendered page is a style sheet applied to the XML data, providing a human readable web page <i>(something I need to get more familiar with)</i>.
<conditions-icon type="forecast-NWS" time-layout="k-p12h-n14-1"> <name>Conditions Icon</name> <icon-link>http://www.nws.noaa.gov/weather/images/fcicons/nbkn.jpg</icon-link> <icon-link>http://www.nws.noaa.gov/weather/images/fcicons/sct.jpg</icon-link> <icon-link>http://www.nws.noaa.gov/weather/images/fcicons/nsct.jpg</icon-link> <icon-link>http://www.nws.noaa.gov/weather/images/fcicons/bkn.jpg</icon-link> <icon-link>http://www.nws.noaa.gov/weather/images/fcicons/nbkn.jpg</icon-link> <icon-link>http://www.nws.noaa.gov/weather/images/fcicons/shra30.jpg</icon-link> <icon-link>http://www.nws.noaa.gov/weather/images/fcicons/nshra30.jpg</icon-link> <icon-link>http://www.nws.noaa.gov/weather/images/fcicons/shra.jpg</icon-link> <icon-link>http://www.nws.noaa.gov/weather/images/fcicons/nshra.jpg</icon-link> <icon-link>http://www.nws.noaa.gov/weather/images/fcicons/shra.jpg</icon-link> <icon-link>http://www.nws.noaa.gov/weather/images/fcicons/nshra.jpg</icon-link> <icon-link>http://www.nws.noaa.gov/weather/images/fcicons/shra.jpg</icon-link> <icon-link>http://www.nws.noaa.gov/weather/images/fcicons/nshra.jpg</icon-link> <icon-link>http://www.nws.noaa.gov/weather/images/fcicons/shra.jpg</icon-link> </conditions-icon>
Design and coding of the file downloads, processing and construction of the status file is the next step. One thought I have, is to download and store locally, all the images. This will keep my page off of their webserver’s request log. External includes of images can be a burden on a remote server, if you start to abuse it. No sense in being a bad netizen. The question that I need to answer is, <i>CAN I</i> get a list of all the potential images (and there are <i>many</i>) and automate the download to system I plan to run this little product upon.
Automation of the XML downloading and parsing, will be handled by PERL (at least in this first tool, it may be converted to JAVA on a second iteration, once I’ve performed usability testing of this first iteration).
Additional binary libraries are required first:
apt-get install libxml2-dev
So, here is the skinny. First, have to move into the location where CPAN unravels the packages:
cd /root/.cpan/build
Next, move into the directory for XML::LibXML and manually build with these steps:
cd XML-LibXML-1.70-OucP9U perl Makefile.PL make make install
Next build the XML::Atom package:
cd XML-Atom-0.37-pcVvAG/ perl Makefile.PL make make install
Next, a few PERL modules must first be installed. NOTE: the XML::SAX modules are suggested, to provide a big boost in XML parsing performance. You might be surprised at how much processing power the typical XML parsing packages require. XML::SAX and XML::SAX::Expat provide a better code for XML parsing.
cpan LWP::UserAgent cpan HTML::TokeParse cpan XML::Feed cpan XML::SAX cpan XML::SAX::Expat cpan Data::Dumper cpan XML::Feed cpan XML::Atom
NOTE:
As hard as I tried, CPAN would not install XML::LibXML nor XML::Atom. So, I had to get down into the muck and manually build the packages. The trick was getting the libxml2 package talked about earlier.
One critical required step, if you are going to write and use custom PERL libraries, you need to add the following line to a shell initialization file. I’ve been using bash, for lack of any reason to use other shells. Thus, I added this line to my ~/.bash_profile
export PERL5LIB=$HOME/plib
Once that is done, you’ll need to source the updated config file, or logout/login to gain the benefits of the change.
The process of finding the desirable data sources, can be a challenge. I already know which of the 5 day forecast sources I want to use.
Maybe I’ll add this one too, just to so I’m reminded of what I’m missing in CA:
Next is a page that contains the CURRENT and HIGH / LOW temps. In the past I have located some text based data files on the NOAA.GOV website. I’ll have to dig up my notes one this. From time to time NOAA moves those files and changes the format, so those notes might not end up being that helpful. It is somewhere to start.
This page here, has a pile of nice detailed information (see compressed screen shot) but does not contain everything I’m looking for:
XML DATA!. XML is something I know more about, than I really care too, but it is a portable, easily parsable and text data format. I’ve not yet investigated the contents here, but it does look promising!
What is a “Weather Reader”? Well, it’s like a “News Reader”, but it’s purpose is to check the web for local weather conditions and display them on a computer screen. For this project I’ve revived an old Averatec laptop, installed Ubuntu 9.10 on it and started to write some PERL.
Where to get the weather data? Personally, I like to go to the source. The public source at NOAA.gov. This is a government agency, paid for by my tax dollars (and yours too if you’re a US tax payer.. of which only 53% of working people in the US are, but that’s another story. So, free for the taking as far as I can tell, and am concerned. Which is good, because the real heavy work is up to me. Finding a URL is one thing, using it for something productive is another thing.
To see what I’m looking at as a source without clicking a bunch of links, here is a screen shot of my indented HTML victim:
Now, the only part that I’m interested in is this one:
I’m sure some of you are asking, “WHAT’S THE POINT!?!?!?”. Well, I have several, not all atop my head. As an avid motorcyclist, weather is almost as import to me as mariners and aviators. Weather might look good for the moment, but around here anything can happen. It’s good to know what the forecast is. And not just in one location. Since I transit two area micro-climates commuting to work, knowing that weather will be at the end points of my journey are are a minimum requirement. Plus, I’m lazy. Not so lazy I won’t write a program to do this, but lazy in the larger sense, of I just want to hit a button and see all the data I want, not have to wait for the Weather Chanel to decided it’s ready to dispense it’s forecast, or hunt around on various local news sites. Nor, frankly, have to click on TWO hyperlinks to see what weather will be like HERE and THERE. Nope, I’m just too lazy for that, so I’ll spend 8-32 hours figuring out a way to save myself 2-3 minutes a day (I didn’t say I’m always smart, but always lazy). Remember; “Laziness is the father of invention!”. Digressing….
The plan is to use PERL, run every 30 minutes from 5 AM-> 1 AM, and grab the weather page shown above. Parse out only that which I want, and then re-construct my own page on my own server. It’s a basically a mashup page of what I want to see.
Checking the source code of the page, I see that, ugh, NOAA did not supply and useful DIV markers that I can use. So, this will be a rel brute-force process to detect, select, extract and re-construct the page of my dreams. Yes, I can be easily entertained.
<table width="100%" border="0" align="center" cellpadding="0" cellspacing="0"> <tr valign ="top" align="center"> <td width="11%"><b>Tonight<br></b><br><img src="/images/wtf/nshra50.jpg" width="55" height="58" alt="Scattered Showers Chance for Measurable Precipitation 50%" title="Scattered Showers Chance for Measurable Precipitation 50%" ><br>Scattered<br>Showers<br>Lo <font color="#0033CC">45 °F</font></td><td width="11%"><b>Tuesday<br></b><br><img src="/images/wtf/shra50.jpg" width="55" height="58" alt="Scattered Showers Chance for Measurable Precipitation 50%" title="Scattered Showers Chance for Measurable Precipitation 50%" ><br>Scattered<br>Showers<br>Hi <font color="#FF0000">54 °F</font></td><td width="11%"><b>Tuesday<br>Night</b><br><img src="/images/wtf/nshra30.jpg" width="55" height="58" alt="Scattered Showers Chance for Measurable Precipitation 30%" title="Scattered Showers Chance for Measurable Precipitation 30%" ><br>Scattered<br>Showers<br>Lo <font color="#0033CC">44 °F</font></td><td width="11%"><b>Wednesday<br></b><br><img src="/images/wtf/bkn.jpg" width="55" height="58" alt="Partly Sunny" title="Partly Sunny" ><br>Partly<br>Sunny<br>Hi <font color="#FF0000">57 °F</font></td><td width="11%"><b>Wednesday<br>Night</b><br><img src="/images/wtf/hi_nshwrs30.jpg" width="55" height="58" alt="Chance Showers Chance for Measurable Precipitation 30%" title="Chance Showers Chance for Measurable Precipitation 30%" ><br>Chance<br>Showers<br>Lo <font color="#0033CC">46 °F</font></td><td width="11%"><b>Thursday<br></b><br><img src="/images/wtf/shra30.jpg" width="55" height="58" alt="Chance Showers Chance for Measurable Precipitation 30%" title="Chance Showers Chance for Measurable Precipitation 30%" ><br>Chance<br>Showers<br>Hi <font color="#FF0000">59 °F</font></td><td width="11%"><b>Thursday<br>Night</b><br><img src="/images/wtf/nshra30.jpg" width="55" height="58" alt="Chance Showers Chance for Measurable Precipitation 30%" title="Chance Showers Chance for Measurable Precipitation 30%" ><br>Chance<br>Showers<br>Lo <font color="#0033CC">46 °F</font></td><td width="11%"><b>Friday<br></b><br><img src="/images/wtf/shra.jpg" width="55" height="58" alt="Chance Showers" title="Chance Showers" ><br>Chance<br>Showers<br>Hi <font color="#FF0000">59 °F</font></td><td width="11%"><b>Friday<br>Night</b><br><img src="/images/wtf/nshra.jpg" width="55" height="58" alt="Chance Showers" title="Chance Showers" ><br>Chance<br>Showers<br>Lo <font color="#0033CC">46 °F</font></td></tr> </table>
Snagging that out of the documents for 2-3 locations will get me what I want. It should look something like this, but the example here does not have any stylesheets applied, it’s a direct rip and mash.
There are a couple of problems with that page though, it’s going to take up too much real estate on the final page. Here, it’s almost too large to view on my laptop’s main screen. So, some embedded CSS is in order. This hear overloads the td type, making any text found within anytd rendered at 80% of the normal size. This also overloads creates a style named headish that I is applied to the section titles:
<style>
.headish {
font-weight:bold;
background-color:#CCC;
font-size: 1.2em;}
td { font-size: 0.8em; }
</style>
The result of that very small change, gives me a smaller visual footprint, and formatting at little more pleasing to my sensibilities:
A little more hammering around in the stylesheet, and adding a CURRENT and High/Low temps forecast for the day. A little closer to the desired mockup. I still need to locate a suitable news feed to process:
While here at home, watching it rain today, I began to wonder what the rainfall trends have been over the last 100 years, and if there is any obvious correlation to the increase or decrease, when compared with the so-called ‘Climate Change’ that is so much in the news.
While poking around the net I came across we chart produced by The Aquinas Project (some sort of research group), but I found it interesting, so I’m posting it here.
What the chart shows, with annual and decade long rainfall trends overlaid is that.. compared to 120 years ago, average rainfall (at least in the search city they chose) is nearly identical.
So, this raised the question for me. Now, I’m not saying I believe that ‘Global Warming’ is happening at all. Frankly, it’s not, and honest good science proves that. BUT, what this chart shows, based on NOAA data (or so they say) that there is NO long-term effect on average rainfall.
So this makes me ask the question:
Assuming Global Warming (or Climate Change.. pick your BS lable) is real, WHO CARES!??! If it’s not affecting the most easily measured weather ‘symptom’, does it really affect humans at all? And let’s be frank here.. we like to say we’re all about nature, but in the end, it’s survival of the human race we are REALLY worried about. Sure, there are some pathological types that aren’t but, who cares about them? I don’t.
So I ask again… if we’re not even seeing more rain.. does it even matter?
I also wonder what the annual U.S. losses dues to weather related natural disasters plot would look like over a similar period (inflation adjusted of course).
Why do I ask that question? Well, because our ‘Great Savior President Obama is promising $100 BILLION dollars of our money to combat this ‘SO CALLED’ problem.
Well, if we’re loosing less than $100 BILLION a decade to natural disasters then I say STOP wasting money trying to prevent them and put that money into damage mitigation!
We were forecasted for snow overnight. Nothing materialized.
Forecast has changed a little bit. Right now it’s raining at just above freezing.
Tomorrow, the streets will probably be a ice rink. I’ll have to get my camera ready to train on the highway outside. It’s steep and heavily traveled. Could be some amusing or frightening stuff caught.
I wish the weather would ‘decide’ to do something other than drizzle on the verge of freezing. That’s boring.