How can we help?


Weather services time is not localised in UI graph

Comments

18 comments

  • Avatar
    Nicholas

    The RainMachine Mixer saves data as UTC (timestamp). Is the mouse over tooltip in the correct day ?

  • Avatar
    Richard Mann (Edited )

    Here is an example of QPF:

    The RainMachine data point is 10 hours different to all the other services. Metno (Blue), OWM (Green) and mine (Orange).

    This is what seems to be happening:

    My data:

          Tuesday, July 24, 2018 12:00:00 AM GMT+10:00 (Date of logging)

          Monday, July 23, 2018 2:00:00 PM (Displayed on graph)

    Mixer:

          Tuesday, July 24, 2018 12:00:00 AM (Displayed on graph)

    If I convert my timestamp and log as UTC , then the graph shows the data point another 10 hours earlier (Monday, July 23, 2018 4:00:00 AM), 20 hours earlier than the mixer point for the same day.

    I just need some guidance here because I feel I am missing something.

    My guess is that the mixer and the parser framework are treating time differently which is why the parsers are consistently out but the mixer is correct (assuming the graph is displayed with UTC on the X-axis).

    If the graph was labelled with either UTC or local time on the X-Axis , this would be more meaningful. Preferably, everything should be graphed in local time.

  • Avatar
    Nicholas

    The mixer always plot data at the start of the day (00:00) AM and it does that by summing each QPF reported in that day for a parser, then summing all QPF totals for parsers in that day and dividing by number of parsers.

    Mixer is day value, parsers are multiple values per day saved at the time they were received (hourly resolution).

  • Avatar
    Richard Mann (Edited )

    Runs at 00:00 in what time zone?

    The problem is that the parser data for the day and the mixer are 10 hours different. If the mixer is looking ahead for the day, then it is using the wrong time period:

     

    ALL the parsers are logging their start day 10 hours earlier than the mixer's time point.

    This conflicts the timeline.

    If the timeline is in UTC, then the local times will be 10 hours earlier. If the mixer runs at midnight local time, then it should log it's time point 10 hours earlier.

    If the timeline is local data, then the parser data is all 10 hours out, but the mixer is correct.

    Which one is it?

  • Avatar
    Nicholas

    Mixer runs hourly, the mixer data is *presented* by API as 00:00 local time. So for 27 Jul, mixer runs and sums all data from 27 and shows it as 27 Jul 00:00, this makes it look as it's earlier in time than parser data. From the graph I see that the mixer follows the data correctly, what you mean is that you got the data from parser that would be 27 Jul 00:00 and it showed as 10h later (not the parser run time but the data from parser) ? 

  • Avatar
    Richard Mann

    We are getting closer! But let's break it down.

    Question 1: Are saying that the x-axis for the weather graph is local time?

    Question 2: Why do the parsers (all, not just mine) first data for the day start 10 hours earlier than midnight?

     

  • Avatar
    Nicholas

    > Question 1: Are saying that the x-axis for the weather graph is local time?

    I don't think web ui does anything special with the data returned from API so it should be shown on "device time zone"

    > Question 2: Why do the parsers (all, not just mine) first data for the day start 10 hours earlier than midnight?

    I'm not entirely sure, I would compare data returned by API (/api/4/parser/<uid>/data) with the graph. Most of our weather services get "daily" data instead of hourly and refresh this constantly so I'll need to watch and try to debug a "hourly" parser to check how it fits the data.

    Does your parser always get the data starting from 00:00 on the day that you run instead of current time at the run ?

     

  • Avatar
    Richard Mann

    My parser gets the date from the data source which is in the local time zone (also specified in the data), which is sometimes the start of day, e.g. forecast rain, and sometimes every 2 hours, e.g. wind.

    The point is that the date is specifically local time, not UTC. I insert with local time, but none of the data in the graph reflects that. it makes it look  like I've inserted it as  UTC, ut I haven't.

    The other point is that all the parsers I use exhibit the same behaviour - Metno, OWM and Wunderground. Maybe there is an issue with me NOT using a US data parser that causes all the data to be out.

    You can see my code at https://github.com/safepay/rainmachine

  • Avatar
    Richard Mann (Edited )

    Had a look at the data for mixer vs parser and I think this is where the graph is wrong.

    Both mixer and parser data has start of day the same exact time stamp and each group of data is neatly segregated into each day period.

    E.g.:

    Mixer:

            {
              "day": "2018-07-25 00:00:00", 
              "temperature": 7.2599999999999998, 
              "rh": 88.019999999999996, 
              "wind": 1.4299999999999999, 
              "solarRad": null, 
              "skyCover": null, 
              "rain": 0.40000000000000002, 
              "et0": null, 
              "pop": null, 
              "qpf": 0.26000000000000001, 
              "condition": 5, 
              "pressure": 100.29000000000001, 
              "dewPoint": 3.96, 
              "minTemp": 2.7200000000000002, 
              "maxTemp": 11.99, 
              "minRH": 75.799999999999997, 
              "maxRH": 90.709999999999994, 
              "et0calc": 0.94999999999999996, 
              "et0final": 0.94999999999999996
            }, 

     

    Parser for the same day - specifically, note that the min/max temps are consistent for each hour as expected:

              "day": "2018-07-25 00:00:00", 
              "hourlyValues": [
                {
                  "hour": "2018-07-25 00:00:00", 
                  "temperature": 5.9000000000000004, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 4.5, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": 0.5, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 01:00:00", 
                  "temperature": 5.4000000000000004, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 4.2000000000000002, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 02:00:00", 
                  "temperature": 5.2000000000000002, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 3.8999999999999999, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 03:00:00", 
                  "temperature": 5.2999999999999998, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 3.7000000000000002, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 04:00:00", 
                  "temperature": 5, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 3.6000000000000001, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 05:00:00", 
                  "temperature": 4.9000000000000004, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.7999999999999998, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 06:00:00", 
                  "temperature": 4.2000000000000002, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.5, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 07:00:00", 
                  "temperature": 3.8999999999999999, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.1000000000000001, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 08:00:00", 
                  "temperature": 4.7000000000000002, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.2000000000000002, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": 0.40000000000000002, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 09:00:00", 
                  "temperature": 6.0999999999999996, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 10:00:00", 
                  "temperature": 7.9000000000000004, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 1.7, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 11:00:00", 
                  "temperature": 9.6999999999999993, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 1.8999999999999999, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 12:00:00", 
                  "temperature": 11.199999999999999, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.7000000000000002, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 13:00:00", 
                  "temperature": 12.300000000000001, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.8999999999999999, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 14:00:00", 
                  "temperature": 12.9, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.8999999999999999, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 15:00:00", 
                  "temperature": 12.9, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.8999999999999999, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 16:00:00", 
                  "temperature": 12.1, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.8999999999999999, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 17:00:00", 
                  "temperature": 10.6, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 3.3999999999999999, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 18:00:00", 
                  "temperature": 8.8000000000000007, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.7999999999999998, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 19:00:00", 
                  "temperature": 7.0999999999999996, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.2000000000000002, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 20:00:00", 
                  "temperature": 6, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.2999999999999998, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 21:00:00", 
                  "temperature": 5.2000000000000002, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.2000000000000002, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 22:00:00", 
                  "temperature": 4.7000000000000002, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 1.8, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 23:00:00", 
                  "temperature": 3.8999999999999999, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 1.8999999999999999, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }
              ]
            }, 

     

    This suggests that the data is correct and that the graph for the mixer is correct but the graph for the parsers is incorrect, with an incorrect time shift due to some time zone confusion.

    All data, including the mixer, seem to be logged in local time, not UTC. it's just the graph that's wrong.

    You should be able to see this in any time zone outside London, such as California which is GMT -7 compared with Sydney which is GMT +10.

    I would like you to confirm if the above is true or not.

  • Avatar
    Richard Mann (Edited )

    Hi,

    I would appreciate it if this UI behaviour can be confirmed as the bug and that the data is correct, including the mixer data.

    I would have thought that a) This is critical to anyone using the Rainmachine as weather data is at the heart of the system; and b) anyone who wants to contribute code to the Rainmachine system, thereby improving the product.

    Thank you.

  • Avatar
    Nicholas

    Data from API is correct, we are investigating why Web UI graphs it like that, most likely the culprit is a javascript date/time function with local timezone.

  • Avatar
    Michael

    I can also confirm the same thing.  Looks like it is in UTC on the x-axis of the graph that shows weather data in the web interface.

  • Avatar
    Jon Waland

    Further to this -

    metno displays (?) about 18 hours ahead of the actual forecast (18c at 18:00-00:00 on 20th is actually reported as 12:00-18:00 on 21st)

    openweathemap is displaying about 9 hours ahead of forcast - (19c at 06:00 on 21st should probably be 15:00 on 21st)
    Relative humidity looks screwy too:

     

  • Avatar
    Matt Ahlgren

    "All data, including the mixer, seem to be logged in local time, not UTC."

    Richard, were you ever able to get confirmation of the above line? I can't get a definitive answer as the existing parsers in the git repository use UTC, but I'm having trouble with mixer data (from the API) not being in the correct day.

    Considering the weather feature is one of the biggest selling points there is very little developer documentation on how to use it properly. It seems we have to piece together comments on this forum with trial and error. I can't enable the weather-based scheduling on my unit yet because I don't have mixer data I can trust.

  • Avatar
    Richard Mann (Edited )

    Hey Matt,

    The issue is not resolved but the underlying data seems to be correct. So I definitely trust the parser I wrote 100%. (https://github.com/safepay/rainmachine)

    But it has been trial and error as you say.

    When I tried logging the data in UTC, the data was another 10 hours out on the graph which seemed completely wrong.

    So I have settled on logging in local time. This gives underlying data with the correct dates and times. Also, the Rainmachine adjust watering accurately, which is the goal.

    I had terrible trouble with the various time methods in the RainMachine development system. They seem to be incorrectly named. For example, one with UTC in the name logged in local time while a similar named one without UTC logged in UTC time. Very confusing.

    From a development point of view, the system is terrible. Almost no documentation, methods, incorrect graphs, can't upload parsers from Windows. And I can't even log in to this forum using Chrome...

    I only got notification of your post today (19th Feb) when you posted on the 1st. ????

    But, I understand that start-up companies struggle to direct their resources and the developers area is not a high priority. So credit needs to be given where it is due. Having said that, the incorrect graphs are unforgivable in a system that uses weather to adjust watering.

  • Avatar
    Nicholas

    Thanks for the feedback I will direct this message to someone to check and improve. From your observations:

    1. rmGetStartOfDayUtc() and rmTimestampToUtcDateAsString() returns the start of day in device local timezone ? 

    2. Web UI graphs are shown in incorrect timezone

    3. Improve weather services developer documentation.

  • Avatar
    Richard Mann

    Good on you Nicholas.

    Yes, the two UTC named methods return local time.

    I would even additionally like someone to verify that my parser it is correct by someone at RainMachine. This may be of value to you as this parser provides Australian BoM data using a simple JSON API.

  • Avatar
    Jon Waland

    Nicholas  - Did anything ever happen on 2 (graphs are incorrect regarding timezone)?

Please sign in to leave a comment.