How can we help?


Weather services time is not localised in UI graph

Comments

13 comments

  • Avatar
    Nicholas

    The RainMachine Mixer saves data as UTC (timestamp). Is the mouse over tooltip in the correct day ?

  • Avatar
    Richard Mann (Edited )

    Here is an example of QPF:

    The RainMachine data point is 10 hours different to all the other services. Metno (Blue), OWM (Green) and mine (Orange).

    This is what seems to be happening:

    My data:

          Tuesday, July 24, 2018 12:00:00 AM GMT+10:00 (Date of logging)

          Monday, July 23, 2018 2:00:00 PM (Displayed on graph)

    Mixer:

          Tuesday, July 24, 2018 12:00:00 AM (Displayed on graph)

    If I convert my timestamp and log as UTC , then the graph shows the data point another 10 hours earlier (Monday, July 23, 2018 4:00:00 AM), 20 hours earlier than the mixer point for the same day.

    I just need some guidance here because I feel I am missing something.

    My guess is that the mixer and the parser framework are treating time differently which is why the parsers are consistently out but the mixer is correct (assuming the graph is displayed with UTC on the X-axis).

    If the graph was labelled with either UTC or local time on the X-Axis , this would be more meaningful. Preferably, everything should be graphed in local time.

  • Avatar
    Nicholas

    The mixer always plot data at the start of the day (00:00) AM and it does that by summing each QPF reported in that day for a parser, then summing all QPF totals for parsers in that day and dividing by number of parsers.

    Mixer is day value, parsers are multiple values per day saved at the time they were received (hourly resolution).

  • Avatar
    Richard Mann (Edited )

    Runs at 00:00 in what time zone?

    The problem is that the parser data for the day and the mixer are 10 hours different. If the mixer is looking ahead for the day, then it is using the wrong time period:

     

    ALL the parsers are logging their start day 10 hours earlier than the mixer's time point.

    This conflicts the timeline.

    If the timeline is in UTC, then the local times will be 10 hours earlier. If the mixer runs at midnight local time, then it should log it's time point 10 hours earlier.

    If the timeline is local data, then the parser data is all 10 hours out, but the mixer is correct.

    Which one is it?

  • Avatar
    Nicholas

    Mixer runs hourly, the mixer data is *presented* by API as 00:00 local time. So for 27 Jul, mixer runs and sums all data from 27 and shows it as 27 Jul 00:00, this makes it look as it's earlier in time than parser data. From the graph I see that the mixer follows the data correctly, what you mean is that you got the data from parser that would be 27 Jul 00:00 and it showed as 10h later (not the parser run time but the data from parser) ? 

  • Avatar
    Richard Mann

    We are getting closer! But let's break it down.

    Question 1: Are saying that the x-axis for the weather graph is local time?

    Question 2: Why do the parsers (all, not just mine) first data for the day start 10 hours earlier than midnight?

     

  • Avatar
    Nicholas

    > Question 1: Are saying that the x-axis for the weather graph is local time?

    I don't think web ui does anything special with the data returned from API so it should be shown on "device time zone"

    > Question 2: Why do the parsers (all, not just mine) first data for the day start 10 hours earlier than midnight?

    I'm not entirely sure, I would compare data returned by API (/api/4/parser/<uid>/data) with the graph. Most of our weather services get "daily" data instead of hourly and refresh this constantly so I'll need to watch and try to debug a "hourly" parser to check how it fits the data.

    Does your parser always get the data starting from 00:00 on the day that you run instead of current time at the run ?

     

  • Avatar
    Richard Mann

    My parser gets the date from the data source which is in the local time zone (also specified in the data), which is sometimes the start of day, e.g. forecast rain, and sometimes every 2 hours, e.g. wind.

    The point is that the date is specifically local time, not UTC. I insert with local time, but none of the data in the graph reflects that. it makes it look  like I've inserted it as  UTC, ut I haven't.

    The other point is that all the parsers I use exhibit the same behaviour - Metno, OWM and Wunderground. Maybe there is an issue with me NOT using a US data parser that causes all the data to be out.

    You can see my code at https://github.com/safepay/rainmachine

  • Avatar
    Richard Mann (Edited )

    Had a look at the data for mixer vs parser and I think this is where the graph is wrong.

    Both mixer and parser data has start of day the same exact time stamp and each group of data is neatly segregated into each day period.

    E.g.:

    Mixer:

            {
              "day": "2018-07-25 00:00:00", 
              "temperature": 7.2599999999999998, 
              "rh": 88.019999999999996, 
              "wind": 1.4299999999999999, 
              "solarRad": null, 
              "skyCover": null, 
              "rain": 0.40000000000000002, 
              "et0": null, 
              "pop": null, 
              "qpf": 0.26000000000000001, 
              "condition": 5, 
              "pressure": 100.29000000000001, 
              "dewPoint": 3.96, 
              "minTemp": 2.7200000000000002, 
              "maxTemp": 11.99, 
              "minRH": 75.799999999999997, 
              "maxRH": 90.709999999999994, 
              "et0calc": 0.94999999999999996, 
              "et0final": 0.94999999999999996
            }, 

     

    Parser for the same day - specifically, note that the min/max temps are consistent for each hour as expected:

              "day": "2018-07-25 00:00:00", 
              "hourlyValues": [
                {
                  "hour": "2018-07-25 00:00:00", 
                  "temperature": 5.9000000000000004, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 4.5, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": 0.5, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 01:00:00", 
                  "temperature": 5.4000000000000004, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 4.2000000000000002, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 02:00:00", 
                  "temperature": 5.2000000000000002, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 3.8999999999999999, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 03:00:00", 
                  "temperature": 5.2999999999999998, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 3.7000000000000002, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 04:00:00", 
                  "temperature": 5, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 3.6000000000000001, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 05:00:00", 
                  "temperature": 4.9000000000000004, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.7999999999999998, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 06:00:00", 
                  "temperature": 4.2000000000000002, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.5, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 07:00:00", 
                  "temperature": 3.8999999999999999, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.1000000000000001, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 08:00:00", 
                  "temperature": 4.7000000000000002, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.2000000000000002, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": 0.40000000000000002, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 09:00:00", 
                  "temperature": 6.0999999999999996, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 10:00:00", 
                  "temperature": 7.9000000000000004, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 1.7, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 11:00:00", 
                  "temperature": 9.6999999999999993, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 1.8999999999999999, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 12:00:00", 
                  "temperature": 11.199999999999999, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.7000000000000002, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 13:00:00", 
                  "temperature": 12.300000000000001, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.8999999999999999, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 14:00:00", 
                  "temperature": 12.9, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.8999999999999999, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 15:00:00", 
                  "temperature": 12.9, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.8999999999999999, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 16:00:00", 
                  "temperature": 12.1, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.8999999999999999, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 17:00:00", 
                  "temperature": 10.6, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 3.3999999999999999, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 18:00:00", 
                  "temperature": 8.8000000000000007, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.7999999999999998, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 19:00:00", 
                  "temperature": 7.0999999999999996, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.2000000000000002, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 20:00:00", 
                  "temperature": 6, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.2999999999999998, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 21:00:00", 
                  "temperature": 5.2000000000000002, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 2.2000000000000002, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 22:00:00", 
                  "temperature": 4.7000000000000002, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 1.8, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }, 
                {
                  "hour": "2018-07-25 23:00:00", 
                  "temperature": 3.8999999999999999, 
                  "minTemperature": 3, 
                  "maxTemperature": 13, 
                  "rh": null, 
                  "minRh": null, 
                  "maxRh": null, 
                  "wind": 1.8999999999999999, 
                  "solarRad": null, 
                  "skyCover": null, 
                  "rain": null, 
                  "et0": null, 
                  "pop": null, 
                  "qpf": null, 
                  "condition": null, 
                  "pressure": null, 
                  "dewPoint": null
                }
              ]
            }, 

     

    This suggests that the data is correct and that the graph for the mixer is correct but the graph for the parsers is incorrect, with an incorrect time shift due to some time zone confusion.

    All data, including the mixer, seem to be logged in local time, not UTC. it's just the graph that's wrong.

    You should be able to see this in any time zone outside London, such as California which is GMT -7 compared with Sydney which is GMT +10.

    I would like you to confirm if the above is true or not.

  • Avatar
    Richard Mann (Edited )

    Hi,

    I would appreciate it if this UI behaviour can be confirmed as the bug and that the data is correct, including the mixer data.

    I would have thought that a) This is critical to anyone using the Rainmachine as weather data is at the heart of the system; and b) anyone who wants to contribute code to the Rainmachine system, thereby improving the product.

    Thank you.

  • Avatar
    Nicholas

    Data from API is correct, we are investigating why Web UI graphs it like that, most likely the culprit is a javascript date/time function with local timezone.

  • Avatar
    Michael

    I can also confirm the same thing.  Looks like it is in UTC on the x-axis of the graph that shows weather data in the web interface.

  • Avatar
    Jon Waland

    Further to this -

    metno displays (?) about 18 hours ahead of the actual forecast (18c at 18:00-00:00 on 20th is actually reported as 12:00-18:00 on 21st)

    openweathemap is displaying about 9 hours ahead of forcast - (19c at 06:00 on 21st should probably be 15:00 on 21st)
    Relative humidity looks screwy too:

     

Please sign in to leave a comment.