Australia's Bureau of Meteorology
A customer has asked us about more current meteorological data from Australia's Bureau of Meteorology and this is what we did, as an example that can be expanded on:
We looked at the data that is obtainable from http://www.bom.gov.au/ (public data without subscription) we see a few problems:
-
There isn't a service where you could obtain data based on latitude/longitude. Usually most weather services accept a query url like
(http://graphical.weather.gov/xml/sample_products/browser_interface/ndfdXMLclient.php?lat=37.07264&lon=-121.816768&startDate=2016-09-14&numDays=6&Unit=e) -
The closest service that we could find to return data in a computer parsable format is:ftp://ftp.bom.gov.au/anon/gen/fwo/IDQ11295.xml
From: http://reg.bom.gov.au/catalogue/data-feeds.shtml#precis
Queensland Region, Precis XML -
This XML file is quite big (600KB) and it's only for a region.
That being said we've made a startup example for this weather service here: https://github.com/sprinkler/rainmachine-developer-resources/blob/master/sdk-parsers/RMParserFramework/parsers/australia-bom.py
It will get the min,max, qpf avg forecast for a town (defined in params) for Queensland region
If you wish to extend and improve this example code please take a look here: http://wikipage.rainmachine.com/index.php?title=Developing_with_RainMachine_SDK
To install this example code on your RainMachine,:
1. Download the code: https://raw.githubusercontent.com/sprinkler/rainmachine-developer-resources/master/sdk-parsers/RMParserFramework/parsers/australia-bom.py
2. Install it to RainMachine by following this tutorial: http://wikipage.rainmachine.com/index.php?title=How_to_upload_a_new_weather_data_source_parser
-
Hi,
I am looking to buy the Rainmachine Touch HD-16 because I live on a remote property with tank water and need variable control for each area - trees, vegetables, garden, etc.
I am impressed that specific Python scripts can be written and loaded and this sets the Rainmachine apart.
There is real-time data available via a JSON feed. If you can combine the two feeds then you will have a service that suits the whole of Australia.
The user must be able to choose their region and specific station, though, which is quite hard.
For me the two feeds are:
ftp://ftp.bom.gov.au/anon/gen/fwo/IDV10753.xml
(which is the precis XML forecast link for Victoria on this page: http://www.bom.gov.au/catalogue/data-feeds.shtml)
and
http://www.bom.gov.au/fwo/IDV60801/IDV60801.95853.json
(first go to http://www.bom.gov.au/vic/observations/vicall.shtml and find your state / closest station)
(then from the specific station page find the json link below the data E.g. http://www.bom.gov.au/products/IDV60801/IDV60801.95853.shtml)
They are both named "Castlemaine" in the data which seems to be the only link.
This may be a little complex for some users, but if this could be baked in to the system somehow then you would have the only system that gets correct data for all Australian locations.
The alternative is to screenscrape (yes, I know...) the data at http://www.weatherzone.com.au/vic/north-central/castlemaine (for example) which is how the weather add-on for the Kodi PVR works (also written in Python) which is then a one-stop-shop for all Australian weather data.
-
I have written a parser for local observations but I'm unable to properly test it.
It is not clear what historical data is needed and how it is used.
I've written the parser the JSON data from the BoM and to grab the UTC for each historic day (including "today") on the assumption that the data will be consumed properly.
If anyone wants a copy to test/modify let me know.
I would very much like to understand if this can work in conjunction with the "test" BoM parser to see if the Rainmachine get all the data it needs from the two feeds.
-
The recommendation from other forums has been to used more localised third party services that offer API access.
-
Thanks Ben, but that forum thread is over 8 years old and most of the links are invalid. The links and comments also point to the services that I have been using anyway - XML and Json, so I believe I am using the latest local data from the correct source. The alternative is to use a PWS from close by - if one is available. But unfortunately, this cannot be tested in the test suite online.
My basic question is - how would the data I am importing be utilised in Rainmachine? I don't own a Rainmachine and am not keen on buying until I can understand if it will be of use to me. I don't want to buy a device that uses data from 30 kms away, when I have a local bureau weather station 10 kms away.
I have the opportunity to import the data from the BoM using the processes outlined above. I just need to know if that data is valuable to the device.
I.e. Does it use the last hour, day, week? Or does it need daily data for a month?
Is it more useful to get temperature, pressure or rainfall?
Without that understanding, it is hard to build any service.
And if I am to use a PSW, can the RM see one near me or not?
-
Hi Richard,
RainMachines uses 2 types of weather data:
1. Forecasted
2. Observed
RainMachine will always water in the future (using forecast) and corrects with observed data (in the past).
For observed data a single day value for min/max temperature, rain, pressure, dew point, wind is enough. Having more observation types (rain, pressure, wind etc) improves the calculation of EvapoTranspiration (ET). You could fetch observed data by hour but this doesn't makes much sense as we'll calculate an average or max/min depending on observation type. It will mostly waste bandwidth.
Also for observed data we advise to only get data for yesterday as we are saving each day locally on RainMachine database (we keep a history of observations).
As for PSW depends on what service do you use. For example Weather Underground service allows you to select close by stations to be used.
-
Thanks Nicolas, I am currently writing a review of how Smart Irrigation controllers can and could work in Australia. Given that most are US-centric, it is important for me to understand the current support for localised weather data as well as how the systems can be expanded. Once this is complete, I will be doing a hands on test of the units to see how they actually stack up in this market place. The RainMachine appears, on face value, to be a viable unit, noting that I am looking at locations outside capital cities that are not as well services by weather data given the sparse population.
To date, much of the forum discussion here has been centered around Sydney and Melbourne which is not relevant to me.
-
Hi Richard,
Nice, keep us posted with your progress.
We have designed RainMachine to be international from the start but I don't have much experience with bom.gov.au and I'm not sure how reliable it is , as getting data from their services it's quite tedious.
I did some weather queries for Lajamanu and the best seems to be Weather Underground.
-
Hi Eric, Unfortunately I lost the code when a disk failed. So back to square one.
But I think I would actually just use the closest PWS which is the capability that leant me towards Rainmachine in the first place. The Rachio could only do 32 miles away as the closest to where I am in Newstead, Victoria.
-
All, I have written a new parser based on the commercial WillyWeather site that provides an API for the Bureau of Meteorology.
You can view it here: https://github.com/safepay/rainmachine
It handles everything RainMachine needs including:
- Search for nearby weather data and provide station ID
- Entry of preferred local station ID
- Supports the following forecast data:
- Rain amount
- Rain probability
- High/low temperatures
- Hourly temperature
- Hourly wind
- Provides total rainfall for previous day
- Supports home screen weather icons
This can be used as a standalone parser.
While the WillyWeather API is commercial, it is only costing me less than 10 cents a year.
-
Hi all,
I've been extending the existing BoM parser quite a bit including observational data (from the xml feed rather than the json feed as it only requires one feed per state, has more measurement types, and only includes the last observation). What I'm struggling with is information on what observational data RM uses. I've posted this as it's own feed, but there are a few more question on Nicholas's post above:
"For observed data a single day value for min/max temperature, rain, pressure, dew point, wind is enough. Having more observation types (rain, pressure, wind etc) improves the calculation of EvapoTranspiration (ET). You could fetch observed data by hour but this doesn't makes much sense as we'll calculate an average or max/min depending on observation type."
- A 24 hour average for measurements like pressure, dew point, and wind don't make much sense?
- If I was fetching the data by hour, should I fetch the 24 hour rainfall or only the last hour rainfall? How would RM know the difference (as you can only set a timestamp for each data point not a period)?
- Which types are averaged and which are min/maxed?
Also the main question from the other thread is simply which RMWeatherDataType are observation and which are forecast?
-
Also the main question from the other thread is simply which RMWeatherDataType are observation and which are forecast?
All RMWeatherDataTypes can be either observed or forecast there is no restriction.
A 24 hour average for measurements like pressure, dew point,
and wind don't make much sense?It doesn't if you send it only 1 per day and if you send the measured pressure every hour, RainMachine will do that for you.
If I was fetching the data by hour, should I fetch the 24 hour
rainfall or only the last hour rainfall?
How would RM know the difference (as you can only set a
timestamp for each data point not a period) ?In the case of observed data it's best to fetch and send to
RainMachine the rainfall from the beginning of the day
until the observation timestamp
(observation day total rainfall).For forecast data RainMachine knows how to handle hourly QPF values but not for historical/observed.
-
Thanks Nicholas!
So just clarifying:
- RM will only use a daily average for observed data. Can you please clarify which values are averaged vs max/min? I.e. I expect observed temperature is a max/min.
- For observed rainfall it will use the last data point for that day? I assume this is last from the point of view of local time and not UTC?
-
TEMPERATURE = "TEMPERATURE" #[degC]
MINTEMP = "MINTEMP" #[degC]
MAXTEMP = "MAXTEMP" #[degC]
RH = "RH" #[percent]
MINRH = "MINRH" #[percent]
MAXRH = "MAXRH" #[percent]
WIND = "WIND" #[meter/sec]
SOLARRADIATION = "SOLARRADIATION" #[megaJoules / square meter per hour]
SKYCOVER = "SKYCOVER" #[percent]
RAIN = "RAIN" #[mm]
ET0 = "ET0" #[mm]
POP = "POP" #[percent]
QPF = "QPF" #[mm] -
CONDITION = "CONDITION" #[string]
PRESSURE = "PRESSURE" #[kilo Pa]
DEWPOINT = "DEWPOINT" #[degC]MAX/MIN entries will use max(measurements) and min respectively. If you don't send a max/min it will be taken from temperature for example. Same for RH.
For rain it should be the last measurement unless you have another source/parser that sends rain measurements in which case the 2 values will be averaged.
-
BOM Australia have collaborated with Met Office UK to collect weather observations in WOW.
-
I made some changes to use the JSON data source from BOM for observations, and it will automatically determine the nearest weather station that supports rain and temperature (assuming your location is configured in RainMachine of course), but you need to manually configure the state in the add-on. The data feed is every 30 minutes, so it seems to produce some pretty decent data. Note that there are MANY weather stations that collect some data, but not necessarily all data. I filtered the list based on rain/temp as this seemed like the baseline required. Also, weirdly, there are some weather stations that don't provide a data feed. I don't know why, but it's not a large number. I removed those from the available list.
For forecasts, I also changed the forecast retrieval so you can enter a state as well as a forecast location, so it should work for other states as well now (it does for NSW at least).
I basically just wanted something that worked for me, and it seems to so far (and I wanted to avoid needing to run another service to support it).
EDIT: I have just noticed that RainMachine seems to only deal with hourly reports, not less than that. I will have to update it to average out the two 30 minute reports (and more importantly, sum the rain).
-
G'day Nick,
Lat/log is -35.4123 and 149.0936 (Monash, ACT). I have a Bom weather site (Isabella Plains) a few hundred meters away, which is very convenient!
I just logged onto my rainmachine via its local ip adress and checked the weather details. The error is 'error: no data received from server', but when I look through the graphs I can see weather data from that source in the graph tabs for Rain (QPF), Max temp and Min temp, but missing from all the other tabs.
So it appears to be not gathering all the data it needs to calculate the watering amount correctly, maybe?
-
Okay I can reproduce, thanks for the co-ords. For reasons which aren't clear to me, for that specific file, I get a '403 forbidden' response from the script. Using a browser, as you noted, works fine. In the script I explicitly used a user-agent of "curl" which is a command line tool, because that most closely matched what we are doing... however that appears to now be blocked. If I change the user-agent to match a real web browser (say Chrome), it works again.
-
Hi Nick,
Thanks for uploading the new patch for the fix. I've uploaded the new version and now encountering a new error 'Failed to find Forecast Area'.
Once I've uploaded the new file (deleted the old version first), then saved. I then go into the Weather Services and change the 'Forecast Area' from the default Terrey Hills, to State 'NSW' (for some reason Canberra forecast areas are listed as NSW) and Forecast Area to 'Monash' and save. The rainmachine finds the weather station from the list and changes the Forecast Area to 'Tuggeranong' which is correct.
But once saved, refreshed, and then log out and log back in again it comes up with the error 'Failed to find Forecast Area'.
Any ideas?
It seems like a pretty straight forward process to upload the new file and change the forecast area. If I leave the default Forecast Area to Terrey Hills it pick it up fine and loads the data for Terrey Hills.
Please sign in to leave a comment.
Comments
24 comments