Weather Underground public station actual rainfall data --- Did we really need the API?
Since WU disabled the API system allowing us to grab the actual rainfall data from the most local weather station my Rainmachine has been much less useful. It gets the rainfall forecast which tends to be over the actual rainfall, then never waters.
I've noticed that on the web, we can pull what appears to be a CSV file that contains the day's data for a weather station. Would it be possible for the Rainmachine to use this web link, rather than the API to pull down the weather data it used to get from the API?
It seems like something that someone knowledgeable in scripting could setup to vary the weather station name as a variable, change the date as needed, and parse the data returned. Is this something the Rainmachine would be capable of, or can it just outright not get a HTTPS data feed?
If this works, the full previous functionality could be restored to the WU plugin, without the need for an API key. Hopefully I'm right, but if not please let me know why I'm wrong.
Thanks,
Justin
-
Also, the log of my HD-16, I see the following request going to wunderground;
https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID= KKYCREST23&day=16&month=6&year=2019&graphspan=week&format=0&units=metric
Which show the following error: error: HTTP Error 400: Bad Request
However when I enter that same url directly into a browser, I get the following which clearly show observed data for the past 3 days;Date,TemperatureHighC,TemperatureAvgC,TemperatureLowC,DewpointHighC,DewpointAvgC,DewpointLowC,HumidityHigh,HumidityAvg,HumidityLow,PressureMaxhPa,PressureMinhPa,WindSpeedMaxKMH,WindSpeedAvgKMH,GustSpeedMaxKMH,PrecipitationSumCM
2019-6-16,28.6,24.0,19.4,20.0,14.6,6.1,89,63,29,1007,1002,20,1,28,3.76
2019-6-17,28.0,24.3,20.6,20.9,18.0,13.6,96,75,43,1008,1004,13,0,20,1.35
2019-6-18,22.3,21.9,21.6,20.2,19.9,19.7,89,88,88,1007,1006,6,0,8,0.53Probably worth noting that the "KKYCREST2" weather station is not mine. It's just one nearby.
-
I'm not sure if this is the correct way to ask, but can we get an official response from the support team at RainMachine as to why the observed data seems to be available from WUnderground but it is not working on our RainMachines? Is the only way that I can fix this is by spending a bunch of money on a Weather Station?
Users need some way of getting observed weather data into their RainMachine device so that the proper calculations are made. After all, that is why we all bought the device. The NOAA forecasting is just not accurate enough. For instance, the other day NOAA forecasted 1.5" of rain for my area... it didn't rain 1 drop.
RainMachine... Please Help!!
-
The WUnderground service uses the same url that you find out when not configured with a API Key. It still works on our internal test machines although it's not very reliable. It does seem to get errors from time to time, probably related to WUnderground servers.
You can try to REFRESH the weather services by hand and/or select another station to check if you can get data.
We are also working on getting this observed data from NOAA (which has a good part of stations that WUnderground has) and integrate this in the NOAA weather service.
-
That is important into to know, I removed the old API key and saw the same entry as RDLinKY, I added a few options for stations around me, and it seems to be pulling data from one station more consistently. For some reason the station closest to me never returns data even though it all appears there when loading the page in a browser, but the second closest station returns rainfall data so its better than nothing.
I had also found some beta wu parser code on here but it seems to work the same way as the officially integrated code, from what I can tell at least.
Keep up the good work, if you want any specifics or more details let me know.
Thanks
JR -
Can you elaborate on what the problem is with getting NOAA observed data?
From https://w1.weather.gov/xml/current_obs/index.xml there is a like for the weather stations to html_url which has the observation history. The html links include 1/3/6 hr precipitation data. Is there a problem with that field?
-
I found some spare time and made a quick parser for NOAA observed... may/maynot work for others, but seems to have good results from today's rain.
-
Great addition thank you ! We are trying to stay away from HTML parsing and use CDC API data which at this moment doesn't send precipitation data.
I haven't run the parser yet but I would recommend saving just 1 value per day with timestamp set to day timestamp at 00:00. This would prevent issues if the hourly intervals change during successive runs.
-
Understandable you want to avoid scraping HTML :)
I noticed in the "Observed Rain Amount" in the android app after "Show Logs" on the Dashboard Weather pane seems to pull the value for only the last value added for that day rather than an accumulation over the full day.
Can you elaborate on the expectation for how observed data should be entered into the RM database? Will the actual value for observed rain that gets used be an accumulation over the entire day? From a few other posts it sounds like it should be, but most parser only add a single daily value for "yesterday".
Because of how NOAA updates the .html, I don't necessarily have all the data for yesterday when the parser runs at midnight today... Is it safe to update yesterday's daily accumulation multiple times?
Please sign in to leave a comment.
Comments
12 comments