Weather services time is not localised in UI graph
As I have been developing a weather parser I am using the weather services graph to see what my data looks like.
I have data that matches reasonably well, as expected, to the OpenWeatherMap and METNO data.
But, all services are graphed with the timescale set to UTC I believe. This is 10 hours behind my location.
Therefore the min/max temp graphs, which are flat lines that change at start of day, are changing at 2pm the preceding day.
The graph does not appear to take in to account localised time.
The picture below shows the start of day of the parser services circled for the 24th July. Note that the parser data is 10 hours too early.
All the parsers are consistent.
This has made it very difficult to determine whether the data is correct for my parser, which is logging a local timestamp. (If I log with UTC the graph is 20 hours early = 2x10 hours)
It also makes me wonder if the RainMachine is using the timezone correctly when adjusting the watering amount.
-
Here is an example of QPF:
The RainMachine data point is 10 hours different to all the other services. Metno (Blue), OWM (Green) and mine (Orange).
This is what seems to be happening:
My data:
Tuesday, July 24, 2018 12:00:00 AM GMT+10:00 (Date of logging)
Monday, July 23, 2018 2:00:00 PM (Displayed on graph)
Mixer:
Tuesday, July 24, 2018 12:00:00 AM (Displayed on graph)
If I convert my timestamp and log as UTC , then the graph shows the data point another 10 hours earlier (Monday, July 23, 2018 4:00:00 AM), 20 hours earlier than the mixer point for the same day.
I just need some guidance here because I feel I am missing something.
My guess is that the mixer and the parser framework are treating time differently which is why the parsers are consistently out but the mixer is correct (assuming the graph is displayed with UTC on the X-axis).
If the graph was labelled with either UTC or local time on the X-Axis , this would be more meaningful. Preferably, everything should be graphed in local time.
-
The mixer always plot data at the start of the day (00:00) AM and it does that by summing each QPF reported in that day for a parser, then summing all QPF totals for parsers in that day and dividing by number of parsers.
Mixer is day value, parsers are multiple values per day saved at the time they were received (hourly resolution).
-
Runs at 00:00 in what time zone?
The problem is that the parser data for the day and the mixer are 10 hours different. If the mixer is looking ahead for the day, then it is using the wrong time period:
ALL the parsers are logging their start day 10 hours earlier than the mixer's time point.
This conflicts the timeline.
If the timeline is in UTC, then the local times will be 10 hours earlier. If the mixer runs at midnight local time, then it should log it's time point 10 hours earlier.
If the timeline is local data, then the parser data is all 10 hours out, but the mixer is correct.
Which one is it?
-
Mixer runs hourly, the mixer data is *presented* by API as 00:00 local time. So for 27 Jul, mixer runs and sums all data from 27 and shows it as 27 Jul 00:00, this makes it look as it's earlier in time than parser data. From the graph I see that the mixer follows the data correctly, what you mean is that you got the data from parser that would be 27 Jul 00:00 and it showed as 10h later (not the parser run time but the data from parser) ?
-
> Question 1: Are saying that the x-axis for the weather graph is local time?
I don't think web ui does anything special with the data returned from API so it should be shown on "device time zone"
> Question 2: Why do the parsers (all, not just mine) first data for the day start 10 hours earlier than midnight?
I'm not entirely sure, I would compare data returned by API (/api/4/parser/<uid>/data) with the graph. Most of our weather services get "daily" data instead of hourly and refresh this constantly so I'll need to watch and try to debug a "hourly" parser to check how it fits the data.
Does your parser always get the data starting from 00:00 on the day that you run instead of current time at the run ?
-
My parser gets the date from the data source which is in the local time zone (also specified in the data), which is sometimes the start of day, e.g. forecast rain, and sometimes every 2 hours, e.g. wind.
The point is that the date is specifically local time, not UTC. I insert with local time, but none of the data in the graph reflects that. it makes it look like I've inserted it as UTC, ut I haven't.
The other point is that all the parsers I use exhibit the same behaviour - Metno, OWM and Wunderground. Maybe there is an issue with me NOT using a US data parser that causes all the data to be out.
You can see my code at https://github.com/safepay/rainmachine
-
Had a look at the data for mixer vs parser and I think this is where the graph is wrong.
Both mixer and parser data has start of day the same exact time stamp and each group of data is neatly segregated into each day period.
E.g.:
Mixer:
{ "day": "2018-07-25 00:00:00", "temperature": 7.2599999999999998, "rh": 88.019999999999996, "wind": 1.4299999999999999, "solarRad": null, "skyCover": null, "rain": 0.40000000000000002, "et0": null, "pop": null, "qpf": 0.26000000000000001, "condition": 5, "pressure": 100.29000000000001, "dewPoint": 3.96, "minTemp": 2.7200000000000002, "maxTemp": 11.99, "minRH": 75.799999999999997, "maxRH": 90.709999999999994, "et0calc": 0.94999999999999996, "et0final": 0.94999999999999996 },Parser for the same day - specifically, note that the min/max temps are consistent for each hour as expected:
"day": "2018-07-25 00:00:00", "hourlyValues": [ { "hour": "2018-07-25 00:00:00", "temperature": 5.9000000000000004, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 4.5, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": 0.5, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 01:00:00", "temperature": 5.4000000000000004, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 4.2000000000000002, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 02:00:00", "temperature": 5.2000000000000002, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 3.8999999999999999, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 03:00:00", "temperature": 5.2999999999999998, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 3.7000000000000002, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 04:00:00", "temperature": 5, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 3.6000000000000001, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 05:00:00", "temperature": 4.9000000000000004, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 2.7999999999999998, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 06:00:00", "temperature": 4.2000000000000002, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 2.5, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 07:00:00", "temperature": 3.8999999999999999, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 2.1000000000000001, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 08:00:00", "temperature": 4.7000000000000002, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 2.2000000000000002, "solarRad": null, "skyCover": null, "rain": 0.40000000000000002, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 09:00:00", "temperature": 6.0999999999999996, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 2, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 10:00:00", "temperature": 7.9000000000000004, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 1.7, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 11:00:00", "temperature": 9.6999999999999993, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 1.8999999999999999, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 12:00:00", "temperature": 11.199999999999999, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 2.7000000000000002, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 13:00:00", "temperature": 12.300000000000001, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 2.8999999999999999, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 14:00:00", "temperature": 12.9, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 2.8999999999999999, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 15:00:00", "temperature": 12.9, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 2.8999999999999999, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 16:00:00", "temperature": 12.1, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 2.8999999999999999, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 17:00:00", "temperature": 10.6, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 3.3999999999999999, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 18:00:00", "temperature": 8.8000000000000007, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 2.7999999999999998, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 19:00:00", "temperature": 7.0999999999999996, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 2.2000000000000002, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 20:00:00", "temperature": 6, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 2.2999999999999998, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 21:00:00", "temperature": 5.2000000000000002, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 2.2000000000000002, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 22:00:00", "temperature": 4.7000000000000002, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 1.8, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null }, { "hour": "2018-07-25 23:00:00", "temperature": 3.8999999999999999, "minTemperature": 3, "maxTemperature": 13, "rh": null, "minRh": null, "maxRh": null, "wind": 1.8999999999999999, "solarRad": null, "skyCover": null, "rain": null, "et0": null, "pop": null, "qpf": null, "condition": null, "pressure": null, "dewPoint": null } ] },This suggests that the data is correct and that the graph for the mixer is correct but the graph for the parsers is incorrect, with an incorrect time shift due to some time zone confusion.
All data, including the mixer, seem to be logged in local time, not UTC. it's just the graph that's wrong.
You should be able to see this in any time zone outside London, such as California which is GMT -7 compared with Sydney which is GMT +10.
I would like you to confirm if the above is true or not.
-
Hi,
I would appreciate it if this UI behaviour can be confirmed as the bug and that the data is correct, including the mixer data.
I would have thought that a) This is critical to anyone using the Rainmachine as weather data is at the heart of the system; and b) anyone who wants to contribute code to the Rainmachine system, thereby improving the product.
Thank you.
-
Further to this -
metno displays (?) about 18 hours ahead of the actual forecast (18c at 18:00-00:00 on 20th is actually reported as 12:00-18:00 on 21st)
openweathemap is displaying about 9 hours ahead of forcast - (19c at 06:00 on 21st should probably be 15:00 on 21st)
Relative humidity looks screwy too:
-
"All data, including the mixer, seem to be logged in local time, not UTC."
Richard, were you ever able to get confirmation of the above line? I can't get a definitive answer as the existing parsers in the git repository use UTC, but I'm having trouble with mixer data (from the API) not being in the correct day.
Considering the weather feature is one of the biggest selling points there is very little developer documentation on how to use it properly. It seems we have to piece together comments on this forum with trial and error. I can't enable the weather-based scheduling on my unit yet because I don't have mixer data I can trust.
-
Hey Matt,
The issue is not resolved but the underlying data seems to be correct. So I definitely trust the parser I wrote 100%. (https://github.com/safepay/rainmachine)
But it has been trial and error as you say.
When I tried logging the data in UTC, the data was another 10 hours out on the graph which seemed completely wrong.
So I have settled on logging in local time. This gives underlying data with the correct dates and times. Also, the Rainmachine adjust watering accurately, which is the goal.
I had terrible trouble with the various time methods in the RainMachine development system. They seem to be incorrectly named. For example, one with UTC in the name logged in local time while a similar named one without UTC logged in UTC time. Very confusing.
From a development point of view, the system is terrible. Almost no documentation, methods, incorrect graphs, can't upload parsers from Windows. And I can't even log in to this forum using Chrome...
I only got notification of your post today (19th Feb) when you posted on the 1st. ????
But, I understand that start-up companies struggle to direct their resources and the developers area is not a high priority. So credit needs to be given where it is due. Having said that, the incorrect graphs are unforgivable in a system that uses weather to adjust watering.
-
Thanks for the feedback I will direct this message to someone to check and improve. From your observations:
1. rmGetStartOfDayUtc() and rmTimestampToUtcDateAsString() returns the start of day in device local timezone ?
2. Web UI graphs are shown in incorrect timezone
3. Improve weather services developer documentation.
-
Hi all,
Any update on this?
Using the Web UI Version 1.13, and the default parser for Forecast.IO and Mixer I see the following:
The Forecast.IO is showing the daily QPF at 8 AUG 23h:00 with an amount of 3.52mm
The RainMachine Mixer is showing the daily QPF at 9 AUG 00h:00
My current timezone is WEST (Western European Summer Time) UTC/GMT +1 hour
Collecting the data from the API for the ForecastIO:
{
"day": "2019-08-09 00:00:00",
"hourlyValues": [
{
"hour": "2019-08-09 00:00:00",
"temperature": null,
"minTemperature": 17.99,
"maxTemperature": 22.68,
"rh": 86,
"minRh": 86,
"maxRh": 86,
"wind": 3.83,
"solarRad": null,
"skyCover": null,
"rain": null,
"et0": null,
"pop": 38,
"qpf": 3.52,
"condition": 19,
"pressure": null,
"dewPoint": 17.67
}
]
}Regards,
Pedro
-
Mixer always shows data at 00:00, the weather services like forecast.io display data with the date/time that was included in report. There is also an issue with graph rendering with the library that we use. Although we disabled UTC conversion and forced the library to work only with local time, some browsers (even same browser different versions) show this correctly and some show it with UTC offset. For example same graph looks correctly when I open on my iPhone Safari and looks shifted when I open on a MacOS Chrome (although it was used to be displayed correctly in Chrome). Both iPhone and Mac have the same timezone setup in settings.
We are still looking for ways to improve this, library was updated in latest update and it showed correctly on our test machines but we know that there are still cases where it doesn't.
-
> some browsers (even same browser different versions) show this correctly and some show it with UTC offset. For example same graph looks correctly when I open on my iPhone Safari and looks shifted when I open on a MacOS Chrome (although it was used to be displayed correctly in Chrome). Both iPhone and Mac have the same timezone setup in settings.
I concur with this assessment, the weird UTC behavior seems limited to viewing the Web GUI via desktop (e.g. Windows, MacOS) browsers like Chrome:
Please sign in to leave a comment.
Comments
24 comments