Local Observed Rain seems to be ignored?

As the rain forecast is often incorrect for my location (a lot more rain in the forecast than how much actually comes down), I do a local weather push of the observed amount through the API (which generally seems to work well). But for whatever reason only the forecast rain amount seems to be used for calculations and not the observed amount. Any idea what could go wrong?

As an example there has been 0.84 mm of rain observed combined on 9/28, 9/29 and 9/30:

But the logs for a run of "Schotterrasen" on 10/1 at 3:00 am show 2.55 mm "Rain since last run" (last run was on 9/29 at 3:00 am also):

Below is the log just before the program ran on 1/10. I am happy to provide more log, I just did not wand to make the post too large.

2020-09-30 22:59:57,435 - INFO  - rmHTTPResourceApi4Parser:416 - Local Weather API push: 1601499597: {'rain': 0.27939999999999998}
2020-09-30 23:14:08,258 - INFO  - rmMixer:123 - *** DATA IN THE PAST: 2020-09-01 00:00:00
2020-09-30 23:14:08,268 - INFO  - rmMixer:124 - *** Received data: ('2020-09-01 00:00:00', temp=8.2699999999999996/1, minTemp=None/0, maxTemp=None/0, rh=30.25/1, minRh=None/0, maxRh=None/0, wind=1.6799999999999999/1, solarRad=None/0, skyCover=None/0, rain=None/0, et0=None/0, pop=None/0, qpf=0/1, condition=3/0, pressure=50.859999999999999/1, dewPoint=4.3200000000000003/1, userData=None, minTemp=11.699999999999999/0, maxTemp=18.899999999999999/0, minRH=47.600000000000001/0, maxRH=83.430000000000007/0, et0cal=None, et0final=None)
2020-09-30 23:14:08,315 - INFO  - rmMixer:131 - *** Added Existing data: '2020-09-01 00:00:00', temp=8.2699999999999996/1, minTemp=None/0, maxTemp=None/0, rh=30.25/1, minRh=None/0, maxRh=None/0, wind=0.56000000000000005/1, solarRad=None/0, skyCover=None/0, rain=None/0, et0=None/0, pop=None/0, qpf=0/1, condition=3/1, pressure=50.859999999999999/1, dewPoint=4.3200000000000003/1, userData=None, minTemp=11.699999999999999/1, maxTemp=18.899999999999999/1, minRH=47.600000000000001/1, maxRH=83.430000000000007/1, et0cal=2.744178419100217, et0final=2.7400000000000002
2020-09-30 23:14:08,329 - INFO  - rmMixer:132 - *** All available Data: [('2020-09-01 00:00:00', temp=8.2699999999999996/1, minTemp=None/0, maxTemp=None/0, rh=30.25/1, minRh=None/0, maxRh=None/0, wind=1.6799999999999999/1, solarRad=None/0, skyCover=None/0, rain=None/0, et0=None/0, pop=None/0, qpf=0/1, condition=3/0, pressure=50.859999999999999/1, dewPoint=4.3200000000000003/1, userData=None, minTemp=11.699999999999999/0, maxTemp=18.899999999999999/0, minRH=47.600000000000001/0, maxRH=83.430000000000007/0, et0cal=None, et0final=None), ('2020-09-01 00:00:00', temp=8.2699999999999996/1, minTemp=None/0, maxTemp=None/0, rh=30.25/1, minRh=None/0, maxRh=None/0, wind=0.56000000000000005/1, solarRad=None/0, skyCover=None/0, rain=None/0, et0=None/0, pop=None/0, qpf=0/1, condition=3/1, pressure=50.859999999999999/1, dewPoint=4.3200000000000003/1, userData=None, minTemp=11.699999999999999/1, maxTemp=18.899999999999999/1, minRH=47.600000000000001/1, maxRH=83.430000000000007/1, et0cal=2.744178419100217, et0final=2.7400000000000002)]
2020-09-30 23:14:12,071 - INFO  - rmSimulator:614 - * Correction: Zone 5 et0Delta=-0.09, qpfDelta=-1.61, rain=0.84, realPrecip=0.84, qpfUsedLimited=2.45
2020-09-30 23:14:18,493 - INFO  - rmSimulator:614 - * Correction: Zone 2 et0Delta=0.87, qpfDelta=-4.82, rain=0.28, realPrecip=0.28, qpfUsedLimited=5.10
2020-09-30 23:14:18,515 - INFO  - rmSimulator:614 - * Correction: Zone 3 et0Delta=0.87, qpfDelta=-4.82, rain=0.28, realPrecip=0.28, qpfUsedLimited=5.10
2020-09-30 23:14:18,542 - INFO  - rmSimulator:614 - * Correction: Zone 4 et0Delta=0.87, qpfDelta=-4.82, rain=0.28, realPrecip=0.28, qpfUsedLimited=5.10
2020-09-30 23:59:57,548 - INFO  - rmHTTPResourceApi4Parser:416 - Local Weather API push: 1601503197: {'rain': 0.27939999999999998}
2020-10-01 00:14:56,969 - INFO  - rmMixer:123 - *** DATA IN THE PAST: 2020-09-30 00:00:00
2020-10-01 00:14:56,979 - INFO  - rmMixer:124 - *** Received data: ('2020-09-30 00:00:00', temp=None/0, minTemp=None/0, maxTemp=None/0, rh=None/0, minRh=None/0, maxRh=None/0, wind=None/0, solarRad=None/0, skyCover=None/0, rain=0.28000000000000003/1, et0=None/0, pop=None/0, qpf=None/0, condition=None/0, pressure=None/0, dewPoint=None/0, userData=None, minTemp=None/0, maxTemp=None/0, minRH=None/0, maxRH=None/0, et0cal=None, et0final=None)
2020-10-01 00:14:57,001 - INFO  - rmMixer:131 - *** Added Existing data: '2020-09-30 00:00:00', temp=14.619999999999999/1, minTemp=None/0, maxTemp=None/0, rh=85.730000000000004/1, minRh=None/0, maxRh=None/0, wind=0.84499999999999997/1, solarRad=None/0, skyCover=None/0, rain=0.28000000000000003/1, et0=None/0, pop=None/0, qpf=0/1, condition=3/1, pressure=101.3/1, dewPoint=11.529999999999999/1, userData=None, minTemp=11.35/1, maxTemp=17.100000000000001/1, minRH=67.519999999999996/1, maxRH=98.950000000000003/1, et0cal=1.4306820679243524, et0final=1.4299999999999999
2020-10-01 00:14:57,015 - INFO  - rmMixer:132 - *** All available Data: [('2020-09-30 00:00:00', temp=None/0, minTemp=None/0, maxTemp=None/0, rh=None/0, minRh=None/0, maxRh=None/0, wind=None/0, solarRad=None/0, skyCover=None/0, rain=0.28000000000000003/1, et0=None/0, pop=None/0, qpf=None/0, condition=None/0, pressure=None/0, dewPoint=None/0, userData=None, minTemp=None/0, maxTemp=None/0, minRH=None/0, maxRH=None/0, et0cal=None, et0final=None), ('2020-09-30 00:00:00', temp=14.619999999999999/1, minTemp=None/0, maxTemp=None/0, rh=85.730000000000004/1, minRh=None/0, maxRh=None/0, wind=0.84499999999999997/1, solarRad=None/0, skyCover=None/0, rain=0.28000000000000003/1, et0=None/0, pop=None/0, qpf=0/1, condition=3/1, pressure=101.3/1, dewPoint=11.529999999999999/1, userData=None, minTemp=11.35/1, maxTemp=17.100000000000001/1, minRH=67.519999999999996/1, maxRH=98.950000000000003/1, et0cal=1.4306820679243524, et0final=1.4299999999999999)]
2020-10-01 00:15:01,491 - INFO  - rmSimulator:614 - * Correction: Zone 5 et0Delta=-0.09, qpfDelta=-1.61, rain=0.84, realPrecip=0.84, qpfUsedLimited=2.45
2020-10-01 00:15:07,952 - INFO  - rmSimulator:614 - * Correction: Zone 2 et0Delta=0.87, qpfDelta=-4.82, rain=0.28, realPrecip=0.28, qpfUsedLimited=5.10
2020-10-01 00:15:07,969 - INFO  - rmSimulator:614 - * Correction: Zone 3 et0Delta=0.87, qpfDelta=-4.82, rain=0.28, realPrecip=0.28, qpfUsedLimited=5.10
2020-10-01 00:15:07,991 - INFO  - rmSimulator:614 - * Correction: Zone 4 et0Delta=0.87, qpfDelta=-4.82, rain=0.28, realPrecip=0.28, qpfUsedLimited=5.10
2020-10-01 00:59:57,666 - INFO  - rmHTTPResourceApi4Parser:416 - Local Weather API push: 1601506797: {'rain': 0.0}
2020-10-01 01:15:47,832 - INFO  - rmMixer:123 - *** DATA IN THE PAST: 2020-09-30 00:00:00
2020-10-01 01:15:47,846 - INFO  - rmMixer:124 - *** Received data: ('2020-09-30 00:00:00', temp=12.33/1, minTemp=None/0, maxTemp=None/0, rh=85.730000000000004/1, minRh=None/0, maxRh=None/0, wind=2.0699999999999998/1, solarRad=None/0, skyCover=None/0, rain=None/0, et0=None/0, pop=None/0, qpf=0/1, condition=3/0, pressure=101.28/1, dewPoint=10.1/1, userData=None, minTemp=9.1999999999999993/0, maxTemp=17.199999999999999/0, minRH=67.519999999999996/0, maxRH=98.950000000000003/0, et0cal=None, et0final=None)
2020-10-01 01:15:47,867 - INFO  - rmMixer:131 - *** Added Existing data: '2020-09-30 00:00:00', temp=14.619999999999999/1, minTemp=None/0, maxTemp=None/0, rh=85.730000000000004/1, minRh=None/0, maxRh=None/0, wind=0.79000000000000004/1, solarRad=None/0, skyCover=None/0, rain=0.28000000000000003/1, et0=None/0, pop=None/0, qpf=0/1, condition=3/1, pressure=101.3/1, dewPoint=11.529999999999999/1, userData=None, minTemp=11.35/1, maxTemp=17.100000000000001/1, minRH=67.519999999999996/1, maxRH=98.950000000000003/1, et0cal=1.4306820679243524, et0final=1.4299999999999999
2020-10-01 01:15:47,883 - INFO  - rmMixer:132 - *** All available Data: [('2020-09-30 00:00:00', temp=12.33/1, minTemp=None/0, maxTemp=None/0, rh=85.730000000000004/1, minRh=None/0, maxRh=None/0, wind=2.0699999999999998/1, solarRad=None/0, skyCover=None/0, rain=None/0, et0=None/0, pop=None/0, qpf=0/1, condition=3/0, pressure=101.28/1, dewPoint=10.1/1, userData=None, minTemp=9.1999999999999993/0, maxTemp=17.199999999999999/0, minRH=67.519999999999996/0, maxRH=98.950000000000003/0, et0cal=None, et0final=None), ('2020-09-30 00:00:00', temp=14.619999999999999/1, minTemp=None/0, maxTemp=None/0, rh=85.730000000000004/1, minRh=None/0, maxRh=None/0, wind=0.79000000000000004/1, solarRad=None/0, skyCover=None/0, rain=0.28000000000000003/1, et0=None/0, pop=None/0, qpf=0/1, condition=3/1, pressure=101.3/1, dewPoint=11.529999999999999/1, userData=None, minTemp=11.35/1, maxTemp=17.100000000000001/1, minRH=67.519999999999996/1, maxRH=98.950000000000003/1, et0cal=1.4306820679243524, et0final=1.4299999999999999)]
2020-10-01 01:15:50,984 - INFO  - rmSimulator:614 - * Correction: Zone 5 et0Delta=-0.09, qpfDelta=-1.61, rain=0.84, realPrecip=0.84, qpfUsedLimited=2.45
2020-10-01 01:15:57,133 - INFO  - rmSimulator:614 - * Correction: Zone 2 et0Delta=0.87, qpfDelta=-4.82, rain=0.28, realPrecip=0.28, qpfUsedLimited=5.10
2020-10-01 01:15:57,361 - INFO  - rmSimulator:614 - * Correction: Zone 3 et0Delta=0.87, qpfDelta=-4.82, rain=0.28, realPrecip=0.28, qpfUsedLimited=5.10
2020-10-01 01:15:57,378 - INFO  - rmSimulator:614 - * Correction: Zone 4 et0Delta=0.87, qpfDelta=-4.82, rain=0.28, realPrecip=0.28, qpfUsedLimited=5.10
2020-10-01 01:59:57,771 - INFO  - rmHTTPResourceApi4Parser:416 - Local Weather API push: 1601510397: {'rain': 0.0}
2020-10-01 02:16:36,798 - INFO  - rmMixer:123 - *** DATA IN THE PAST: 2020-09-30 00:00:00
2020-10-01 02:16:36,808 - INFO  - rmMixer:124 - *** Received data: ('2020-09-30 00:00:00', temp=12.33/1, minTemp=None/0, maxTemp=None/0, rh=85.730000000000004/1, minRh=None/0, maxRh=None/0, wind=2.0699999999999998/1, solarRad=None/0, skyCover=None/0, rain=None/0, et0=None/0, pop=None/0, qpf=0/1, condition=3/0, pressure=101.28/1, dewPoint=10.1/1, userData=None, minTemp=9.1999999999999993/0, maxTemp=17.199999999999999/0, minRH=67.519999999999996/0, maxRH=98.950000000000003/0, et0cal=None, et0final=None)
2020-10-01 02:16:36,831 - INFO  - rmMixer:131 - *** Added Existing data: '2020-09-30 00:00:00', temp=14.619999999999999/1, minTemp=None/0, maxTemp=None/0, rh=85.730000000000004/1, minRh=None/0, maxRh=None/0, wind=0.79000000000000004/1, solarRad=None/0, skyCover=None/0, rain=0.28000000000000003/1, et0=None/0, pop=None/0, qpf=0/1, condition=3/1, pressure=101.3/1, dewPoint=11.529999999999999/1, userData=None, minTemp=11.35/1, maxTemp=17.100000000000001/1, minRH=67.519999999999996/1, maxRH=98.950000000000003/1, et0cal=1.4306820679243524, et0final=1.4299999999999999
2020-10-01 02:16:36,845 - INFO  - rmMixer:132 - *** All available Data: [('2020-09-30 00:00:00', temp=12.33/1, minTemp=None/0, maxTemp=None/0, rh=85.730000000000004/1, minRh=None/0, maxRh=None/0, wind=2.0699999999999998/1, solarRad=None/0, skyCover=None/0, rain=None/0, et0=None/0, pop=None/0, qpf=0/1, condition=3/0, pressure=101.28/1, dewPoint=10.1/1, userData=None, minTemp=9.1999999999999993/0, maxTemp=17.199999999999999/0, minRH=67.519999999999996/0, maxRH=98.950000000000003/0, et0cal=None, et0final=None), ('2020-09-30 00:00:00', temp=14.619999999999999/1, minTemp=None/0, maxTemp=None/0, rh=85.730000000000004/1, minRh=None/0, maxRh=None/0, wind=0.79000000000000004/1, solarRad=None/0, skyCover=None/0, rain=0.28000000000000003/1, et0=None/0, pop=None/0, qpf=0/1, condition=3/1, pressure=101.3/1, dewPoint=11.529999999999999/1, userData=None, minTemp=11.35/1, maxTemp=17.100000000000001/1, minRH=67.519999999999996/1, maxRH=98.950000000000003/1, et0cal=1.4306820679243524, et0final=1.4299999999999999)]
2020-10-01 02:16:40,090 - INFO  - rmSimulator:614 - * Correction: Zone 5 et0Delta=-0.09, qpfDelta=-1.61, rain=0.84, realPrecip=0.84, qpfUsedLimited=2.45
2020-10-01 02:16:46,696 - INFO  - rmSimulator:614 - * Correction: Zone 2 et0Delta=0.87, qpfDelta=-4.82, rain=0.28, realPrecip=0.28, qpfUsedLimited=5.10
2020-10-01 02:16:46,714 - INFO  - rmSimulator:614 - * Correction: Zone 3 et0Delta=0.87, qpfDelta=-4.82, rain=0.28, realPrecip=0.28, qpfUsedLimited=5.10
2020-10-01 02:16:46,741 - INFO  - rmSimulator:614 - * Correction: Zone 4 et0Delta=0.87, qpfDelta=-4.82, rain=0.28, realPrecip=0.28, qpfUsedLimited=5.10
2020-10-01 02:59:55,204 - INFO  - rmHTTPResourceApi4Parser:416 - Local Weather API push: 1601513995: {'rain': 0.0}
2020-10-01 03:00:03,203 - INFO  - rmProgramScheduler:262 - Program 3 added to watering queue and/or watering history
2020-10-01 03:00:03,284 - INFO  - rmProgramSchedulerWatering:250 - Watering - starting zone: (uid=None, pid=3, zid=5, ust='2020-10-01 03:00:00', rst='2020-10-01 03:00:03', ud=600, md=0, rd=0, aw=0.20347915753862633, flag=6, manual=False, running=True, flowClicksDelta=0)
2020-10-01 03:00:04,902 - INFO  - rmProgramSchedulerWatering:295 - UpdateZone available water: 0.203479157539
2020-10-01 03:00:05,137 - INFO  - rmProgramSchedulerWatering:150 - Watering - zone finished: (uid=None, pid=3, zid=5, ust='2020-10-01 03:00:00', rst='2020-10-01 03:00:03', ud=600, md=0, rd=0, aw=0.20347915753862633, flag=6, manual=False, running=False, flowClicksDelta=0)
2020-10-01 03:00:05,367 - INFO  - rmProgramSchedulerWatering:160 - Watering - program finished: programId=3 flag=6

RainMachine Nicholas October 05, 2020 11:00

That's correct forecast is always used for irrigation prediction and observed rain is used as correction. At a glance the observed rain is used to check yesterday (or last time) computed irrigation time. If QPF for yesterday was bigger or less than observed rain it will add/remove the deficit to the next irrigation.

Felix Storm October 05, 2020 15:18

Ok, thanks Nicholas - some follow-up questions:

I had in mind that the whole process of irrigation prediction and how to read the logs in detail had been documented somewhere, but I can't seem to find it anymore - could you point me to the correct page?
To get back to my actual problem: Is there a way to limit the effect of rain forecast QPF for irrigation prediction (to limit problems due to faulty forecasts) but still take observed rain fully into account?
Or would I have to write my own weather forecast parser (or modify an existing one) and just divide QPF by some factor I deem useful?

Thanks,
Felix

RainMachine Nicholas October 05, 2020 17:53

1. I think you are referring to this one: https://support.rainmachine.com/hc/en-us/articles/228620787-RainMachine-watering-simulation-log-format-explained

2. Unfortunately rain sensitivity apply to both rain and qpf, and as you say doing a custom parser might be the only solution.

Felix Storm October 15, 2020 12:33 (Edited October 15, 2020 12:36)

Ok, thanks. I think I have been able to narrow down my specific problem quite a bit: Somehow the rain forecasts from two runs of the same parser seem to get added even though they are actually both full-day-amounts for the same day (background: time zone here is GMT+2 (so Oct 13 22:00 actually means Oct 14 00:00 due to the graph's display problems), today is the 15th and DWD reported 7.7mm for the 14th whereas last night DWD had reported 7.4mm for the 14th which adds up nicely to the mixer's 15.1mm):

Apparently this gets cleaned up later on since I can find it only for yesterday, but not anymore for any earlier day:

Unfortunately It's still a pretty big problem for me since especially yesterday's rain values are very relevant to calculate the amount of time a zone should run today. I will probably solve this issue by at least temporarily reverting back to using METNO only (which unfortunately usually reports significantly less rain than the actual amount - but that still seems to be better than getting forecasts with double the real amount).

But since the DWD parser has become one of the official parsers, it might still be an issue to look into. My guess is that it has to do with the following: The rain figure that the DWD parser uses reports the amount of rain for the past 24h relative to the timestamp that it's attached to. Therefore the DWD parser always sends yesterday's timestamp when adding rain values:

This leads to the situation that a parser run today will still write some rain amount for yesterday and my guess is that the mixer will inadvertently sum it up with the amount that's already there for yesterday - but again, this is just my guess.

There seems to be another probably minor issue with the DWD parser: The DWD timestamps are UTC, so `rmTimestampFromUTCDateAsString` might be the better choice here:

But since DWD is primarily for Germany and Germany always only has GMT+1 or GMT+2, this is probably not really a big issue ;-)

Felix

RainMachine Nicholas October 15, 2020 15:07

I'm not sure the reason for which the DWD reports QPF for yesterday might be some gimmick that I don't understand. Worth sending your observation to author directly as a new Issue here: https://github.com/SgtSeppel/rainmachine-developer-resources

Beside that the code that you highlight doesn't really give you yesterday all the time (-12h) which might be the actual problem.

How can we help?

Local Observed Rain seems to be ignored?

Comments

Didn't find what you were looking for?