No data sent to InfluxDB when PVOutput refuses request

[anoppe]

Today I've increased the interval on the datalogger to 1 minute. As expected the PVOutput refuses 4 of 5 data points. However, I was surprised that the graphs in Grafana (backed by Influx) didn't show a resolution of 1 minute, but HASS did. When reviewing the code, I think I found the spot where this goes wrong:
https://github.com/johanmeijer/grott/blob/master/grottdata.py#L525C20-L525C20

if not pvout_limit.ok_send(definedkey["pvserial"], conf):
    # Will print a line for the refusal in verbose mode (see GrottPvOutLimit at the top)
    return

The return statement causes the method to complete, but the influxDb processing starts at L592 which will never be reached this way.

A way to fix this, is to move everything inside if conf.pvoutput : to a different function. The same might also be a good idea for the other output types to prevent these kinds of bugs.

Thanks!

Example logs:

one without InfluxDB interaction:

- MQTT jsonmsg: 
			 {"device": "******", "time": "2023-10-15T16:14:09", "buffered": "no",
			 "values": {"datalogserial": ""******",", "pvserial": ""******",",
			 "pvstatus": 1, "pvpowerin": 2020, "pv1voltage": 1458, "pv1current": 6,
			 "pv1watt": 883, "pv2voltage": 1440, "pv2current": 8, "pv2watt": 1137,
			 "pvpowerout": 1967, "pvfrequentie": 5005, "pvgridvoltage": 2330,
			 "pvgridcurrent": 9, "pvgridpower": 1967, "pvgridvoltage2": 0,
			 "pvgridcurrent2": 0, "pvgridpower2": 0, "pvgridvoltage3": 0,
			 "pvgridcurrent3": 0, "pvgridpower3": 0, "totworktime": 624429,
			 "pvenergytoday": 38, "pvenergytotal": 241, "epvtotal": 249, "epv1today": 24,
			 "epv1total": 141, "epv2today": 15, "epv2total": 108, "pvtemperature": 253,
			 "pvipmtemperature": 315, "battery1soc": 0}}
	 - Grott MQTT topic used : energy/growatt
	 - MQTT message message sent
	 - PVOut: Update refused for "******", due to time limitation

	 - Growatt packet received:
		  <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('172.22.0.7', 5279), raddr=('192.168.178.140', 23766)>
	 - Data less then minimum record length, data not processed

log with influxdb processing:

- MQTT jsonmsg: 
			 {"device": ""******",", "time": "2023-10-15T16:12:08", "buffered": "no",
			 "values": {"datalogserial": ""******",", "pvserial": ""******",",
			 "pvstatus": 1, "pvpowerin": 2220, "pv1voltage": 1525, "pv1current": 6,
			 "pv1watt": 890, "pv2voltage": 1504, "pv2current": 9, "pv2watt": 1330,
			 "pvpowerout": 2117, "pvfrequentie": 5002, "pvgridvoltage": 2327,
			 "pvgridcurrent": 9, "pvgridpower": 2133, "pvgridvoltage2": 0,
			 "pvgridcurrent2": 0, "pvgridpower2": 0, "pvgridvoltage3": 0,
			 "pvgridcurrent3": 0, "pvgridpower3": 0, "totworktime": 624185,
			 "pvenergytoday": 38, "pvenergytotal": 241, "epvtotal": 249, "epv1today": 24,
			 "epv1total": 141, "epv2today": 15, "epv2total": 108, "pvtemperature": 254,
			 "pvipmtemperature": 315, "battery1soc": 0}}
	 - Grott MQTT topic used : energy/growatt
	 - MQTT message message sent
	 - Grott send data to PVOutput systemid:  101661 for inverter:  "******",
		 -  {'X-Pvoutput-Apikey': '"******",', 'X-Pvoutput-SystemId': '101661'}
		 -  {'d': '20231015', 't': '16:12', 'v2': 211.7, 'v6': 232.7, 'v1': 3800}
	 - Grott PVOutput response: 
		 -  OK 200: Added Status
	 - Grott InfluxDB publihing started
	 - Grott original time :  2023-10-15T16:12:08 adjusted UTC time for influx :  2023-10-15T14:12:08
	 - Grott influxdb jsonmsg: 
			 [{'measurement': '"******",', 'time': '2023-10-15T14:12:08', 'fields':
			 {'datalogserial': '"******",', 'pvserial': '"******",', 'pvstatus': 1,
			 'pvpowerin': 2220, 'pv1voltage': 1525, 'pv1current': 6, 'pv1watt': 890,
			 'pv2voltage': 1504, 'pv2current': 9, 'pv2watt': 1330, 'pvpowerout': 2117,
			 'pvfrequentie': 5002, 'pvgridvoltage': 2327, 'pvgridcurrent': 9,
			 'pvgridpower': 2133, 'pvgridvoltage2': 0, 'pvgridcurrent2': 0,
			 'pvgridpower2': 0, 'pvgridvoltage3': 0, 'pvgridcurrent3': 0, 'pvgridpower3':
			 0, 'totworktime': 624185, 'pvenergytoday': 38, 'pvenergytotal': 241,
			 'epvtotal': 249, 'epv1today': 24, 'epv1total': 141, 'epv2today': 15,
			 'epv2total': 108, 'pvtemperature': 254, 'pvipmtemperature': 315,
			 'battery1soc': 0}}]
	 - Grott write to influxdb v1
	 - Grott extension processing disabled 

	 - Growatt packet received:
		  <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('172.22.0.7', 5279), raddr=('192.168.178.140', 23766)>
	 - Data less then minimum record length, data not processed

[johanmeijer]

Please look at grott.ini variable {pvoutput] section. You can limit the pvoutput processing with:

pvuplimit = 5

I have to rethink this one while stopp Grott processing if an update does not work is not a very bad thing. The inverter / datalogger will buffer and resend the data at a later moment (hopefullly) when everything works well again. I know here are holes in this thought (and the way it is worked out in the code) but at least changing it might need some additional rethinking.

[anoppe]

Please look at grott.ini variable {pvoutput] section. You can limit the pvoutput processing with:
I've tried this, but the issue is not PVOutput rejecting the request, but the code that detects that the new datapoint is still within the 5min window, and will stop processing the datapoint. Please have a look at the pull request linked to this issue: it will detach the pvoutput 'decision' from the influxdb processing.

I think it would help if the grottdata.py is less procedural, but more functional. I've made a start on this in the PR I submitted. Please ask if you have any questions regarding that change.

Meanwhile, I'm running a fork of the repo with this pr-change in it, and now it works as expected: Only send data to PVOutput every 5 min, but it will send all datapoints to influx.

[johanmeijer]

I do not want to argue with you about that the code can be more streamlined and efficient, but overal it is (and stay's) my project and my code!

I started this project with a reason (to get my inverter information to my own home automation tooling and to learn to write python and use new tools like mqtt, grafana, nodered etc). That is still my goal. With the emphasis on learning/writing python and to understand my own coding (and to support other people). Improvements, remarks, tips etc are welcome, rewriting the code (because of restructuring, better looks) does not help me with achieving my goals.

[anoppe]

but overal it is (and stay's) my project and my code!

Yes of course. However, the goal of my PR is not to impose my opinion on your project, it's to fix a bug.

[ZonPV]

Sorry to say that in version 2.7.8 (Grott Proxy) I get this error:

PVOut: Update refused for FPH1B0200D due to time limitation

This is the section in grott.ini

`[PVOutput]
# PVOutput parameters definitions

pvoutput = True
apikey = XXXXXXXXXXXXXXXXXXXXXXXX
# Data upload limit (in minutes)
pvuplimit = 5
systemid = XXXXX`

This is in the log when I start the software

okt 22 16:07:26 raspberrypi grott[2941]:         growattip:                    47.91.67.66
okt 22 16:07:26 raspberrypi grott[2941]:         growattport:                  5279
okt 22 16:07:26 raspberrypi grott[2941]: _PVOutput:
okt 22 16:07:26 raspberrypi grott[2941]:         pvoutput:                     True
okt 22 16:07:26 raspberrypi grott[2941]:         pvdisv1:                      False
okt 22 16:07:26 raspberrypi grott[2941]:         pvtemp:                       False
okt 22 16:07:26 raspberrypi grott[2941]:         pvurl:                        https://pvoutput.org/service/r2/addstatus.jsp
okt 22 16:07:26 raspberrypi grott[2941]:         pvapikey:                     XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
okt 22 16:07:26 raspberrypi grott[2941]:         pvinverters:                  1
okt 22 16:07:26 raspberrypi grott[2941]:         pvsystemid:                   XXXXX`

[johanmeijer]

Because you have specified:

pvuplimit = 5

Grott will sent only one message per 5 minutes to pvoutput. The message in between will be "refused" (=not send).

So probably you will se one successful update per 5 minutes.

[anoppe]

But it is still supposed to send data to InfluxDB, right?

[ZonPV]

Because you have specified:

pvuplimit = 5

Grott will sent only one message per 5 minutes to pvoutput. The message in between will be "refused" (=not send).

So probably you will se one successful update per 5 minutes.

Oke, now understand this message. Thought it was a refuse of PVoutput.org. But now I remember that sends another message.
So probably you will se one successful update per 5 minutes.
Yes, that works perfect.
Thanks for info.

[johanmeijer]

But it is still supposed to send data to InfluxDB, right?

Yes! Isn't that working then? Be aware I haven't wrote the limit part (and maybe haven't test it enough before committed the pr).

I will test it this week.

[anoppe]

No, that is the issue the PR attached to this issue will fix... For completeness, here is the link again: #452

If you want to test, try to reproduce this scenario: set the pvoutput limit to a higher value than the Growatt inverter's interval. In my situation, the growatt inverter interval is set to 1 minute, and pvoutput to 5 minutes.

[anoppe]

Hi @johanmeijer any chance you are going to include this in the codebase? Then I can move away from my fork and use the latest greatest version...

Thanks!