Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No timeout or no answer from coral Google TPU #25

Open
fennec622 opened this issue Dec 13, 2020 · 5 comments
Open

No timeout or no answer from coral Google TPU #25

fennec622 opened this issue Dec 13, 2020 · 5 comments

Comments

@fennec622
Copy link

Hi

i use coral Tpu with mlapi and sometimes

i have no answer from tpu

DEBUG: get_file returned: ./images/52f889a3-9fba-43a0-8b30-17e91aa74cf4.jpg Dec 13 2020 17:43:39.463413 [DBG 1] |---------- TPU (input image: 600w*337h) ----------| Dec 13 2020 17:43:39.463456 [DBG 1] Waiting for TPU lock... Dec 13 2020 17:43:39.463608 [DBG 1] Got TPU lock for detection... Dec 13 2020 17:43:39.478238 [DBG 1] Released TPU detection lock Dec 13 2020 17:43:39.478296 [DBG 1] Coral TPU detection took: 14.678 milliseconds Dec 13 2020 17:43:39.478356 [DBG 2] core model detection over, got 1 objects. Now filtering Dec 13 2020 17:43:39.478384 [DBG 3] Max object size found to be: 70% Dec 13 2020 17:43:39.478427 [DBG 2] Converted 70% to 141540.0 Dec 13 2020 17:43:39.478457 [DBG 2] Returning filtered list of 1 objects. DEBUG: Object Recognition requested DEBUG: get_file returned: ./images/fd0ff2f5-3154-4ce5-965f-6b7b360d8583.jpg Dec 13 2020 17:43:39.527889 [DBG 1] |---------- TPU (input image: 600w*337h) ----------| Dec 13 2020 17:43:39.527935 [DBG 1] Waiting for TPU lock... Dec 13 2020 17:43:39.528090 [DBG 1] Got TPU lock for detection... Dec 13 2020 17:43:39.543669 [DBG 1] Released TPU detection lock Dec 13 2020 17:43:39.543741 [DBG 1] Coral TPU detection took: 15.64 milliseconds Dec 13 2020 17:43:39.543770 [DBG 2] core model detection over, got 0 objects. Now filtering Dec 13 2020 17:43:39.543807 [DBG 3] Max object size found to be: 70% Dec 13 2020 17:43:39.543855 [DBG 2] Converted 70% to 141540.0 Dec 13 2020 17:43:39.543881 [DBG 2] Returning filtered list of 0 objects. DEBUG: Object Recognition requested DEBUG: get_file returned: ./images/b7aeeafe-2de1-46d6-9848-35e4834e8f0d.jpg Dec 13 2020 17:45:57.693730 [DBG 1] |---------- TPU (input image: 600w*337h) ----------| Dec 13 2020 17:45:57.693800 [DBG 1] Waiting for TPU lock... Dec 13 2020 17:45:57.694104 [DBG 1] Got TPU lock for detection...

and zm_event_start continue to send analyse

1507288 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718145 12 "portier" " All" "/var/cache/zoneminder/events/12/2020-12-13/4718145" 1507289 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718145 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718145 1507290 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718145 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718145 1508483 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718187 12 "portier" " All" "/var/cache/zoneminder/events/12/2020-12-13/4718187" 1508484 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718187 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718187 1508485 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718187 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718187 1508897 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718202 12 "portier" " All" "/var/cache/zoneminder/events/12/2020-12-13/4718202" 1508898 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718202 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718202 1508899 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718202 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718202 1512622 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718335 5 "jardin" " All" "/var/cache/zoneminder/events/5/2020-12-13/4718335" 1512623 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718335 5 jardin All /var/cache/zoneminder/events/5/2020-12-13/4718335 1512624 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718335 5 jardin All /var/cache/zoneminder/events/5/2020-12-13/4718335 1512802 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718353 12 "portier" " All" "/var/cache/zoneminder/events/12/2020-12-13/4718353" 1512803 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718353 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718353 1512804 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718353 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718353 1513016 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718374 12 "portier" " All" "/var/cache/zoneminder/events/12/2020-12-13/4718374" 1513017 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718374 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718374 1513018 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718374 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718374 1514023 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718445 5 "jardin" " All" "/var/cache/zoneminder/events/5/2020-12-13/4718445" 1514024 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718445 5 jardin All /var/cache/zoneminder/events/5/2020-12-13/4718445 1514025 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718445 5 jardin All /var/cache/zoneminder/events/5/2020-12-13/4718445 1514116 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718450 12 "portier" " All" "/var/cache/zoneminder/events/12/2020-12-13/4718450" 1514117 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718450 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718450 1514118 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718450 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718450 1514542 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718478 12 "portier" " All" "/var/cache/zoneminder/events/12/2020-12-13/4718478" 1514543 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718478 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718478 1514544 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718478 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718478 1514669 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718489 5 "jardin" " All" "/var/cache/zoneminder/events/5/2020-12-13/4718489" 1514670 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718489 5 jardin All /var/cache/zoneminder/events/5/2020-12-13/4718489 1514671 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718489 5 jardin All /var/cache/zoneminder/events/5/2020-12-13/4718489 1514889 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718507 5 "jardin" " All" "/var/cache/zoneminder/events/5/2020-12-13/4718507" 1514890 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718507 5 jardin All /var/cache/zoneminder/events/5/2020-12-13/4718507 1514891 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718507 5 jardin All /var/cache/zoneminder/events/5/2020-12-13/4718507 1515015 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718517 5 "jardin" "Motion All" "/var/cache/zoneminder/events/5/2020-12-13/4718517" 1515016 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718517 5 jardin Motion All /var/cache/zoneminder/events/5/2020-12-13/4718517 1515017 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718517 5 jardin Motion All /var/cache/zoneminder/events/5/2020-12-13/4718517 1515085 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718522 5 "jardin" " All" "/var/cache/zoneminder/events/5/2020-12-13/4718522" 1515086 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718522 5 jardin All /var/cache/zoneminder/events/5/2020-12-13/4718522 1515087 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718522 5 jardin All /var/cache/zoneminder/events/5/2020-12-13/4718522 1515439 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718549 5 "jardin" " All" "/var/cache/zoneminder/events/5/2020-12-13/4718549" 1515440 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718549 5 jardin All /var/cache/zoneminder/events/5/2020-12-13/4718549 1515441 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718549 5 jardin All /var/cache/zoneminder/events/5/2020-12-13/4718549 1515713 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718571 5 "jardin" " All" "/var/cache/zoneminder/events/5/2020-12-13/4718571" 1515714 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718571 5 jardin All /var/cache/zoneminder/events/5/2020-12-13/4718571 1515715 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718571 5 jardin All /var/cache/zoneminder/events/5/2020-12-13/4718571 1516031 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718603 12 "portier" " All" "/var/cache/zoneminder/events/12/2020-12-13/4718603" 1516032 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718603 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718603 1516033 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718603 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718603 1518568 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718741 12 "portier" " All" "/var/cache/zoneminder/events/12/2020-12-13/4718741" 1518569 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718741 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718741 1518570 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718741 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718741 1523472 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718941 12 "portier" " All" "/var/cache/zoneminder/events/12/2020-12-13/4718941" 1523473 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718941 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718941 1523474 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718941 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718941 1523704 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718954 12 "portier" " All" "/var/cache/zoneminder/events/12/2020-12-13/4718954" 1523705 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718954 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718954 1523706 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718954 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718954 1525097 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4718994 12 "portier" " All" "/var/cache/zoneminder/events/12/2020-12-13/4718994" 1525098 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718994 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718994 1525099 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4718994 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4718994 1530244 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4719355 12 "portier" " All" "/var/cache/zoneminder/events/12/2020-12-13/4719355" 1530245 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4719355 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4719355 1530246 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4719355 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4719355 1530350 ? S 0:00 sh -c '/var/lib/zmeventnotification/bin/zm_event_start.sh' 4719364 12 "portier" " All" "/var/cache/zoneminder/events/12/2020-12-13/4719364" 1530351 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4719364 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4719364 1530352 ? S 0:00 /bin/bash /var/lib/zmeventnotification/bin/zm_event_start.sh 4719364 12 portier All /var/cache/zoneminder/events/12/2020-12-13/4719364

I juste restart mlapi and notification start

I thinks bug arrived when i have a very fast alarm and lot of request to TPU
@pliablepixels
Copy link
Member

Missed responding to this.Do you still have this issue?
I've found the coral to be stable after I changed the cable and bought a new USB 3.1 cable

@fennec622
Copy link
Author

Ok thanks I try

For the moment I restart all 5 mins with cron

@fennec622
Copy link
Author

i change Server so I make new installation
And change with cable USBC 3.1

Same problem no answer after lot of request in little time

So I apply script for surv and restart

#!/bin/bash if [[ $(ps -ef | grep zm_event_start.sh | grep -v grep | wc -l | awk '{print $1}') -ge 10 ]] then /usr/sbin/service mlapi restart fi

@apsilva
Copy link

apsilva commented Nov 21, 2024

Hi,

i think i'm facing the same issue.. mlapi stop detecting objects and for what i could see is when alot of alarms are fired. Mostly at night, in my setup i can see lot of alarms cause by the spider webs on the cameras... (btw do you know how to scare them? :) ) that triggers lots of zm_detect that get stuck in the system.

In zmeventnotification it was introduce a new ml_timeout parameter that solves it, but now i still need to manually restart mlapi service to re-activate object detection. I guess that i could just restart mlapi.

The last message from mlapi.log before the problem, the last message seams to be waiting to release the lock on "pyzm_uid0_tpu_lock"

Nov 21 2024 02:50:18.508542 [DBG 1] Monitor ID not specified, or not found in mlapi config, using zm_detect overrides
Nov 21 2024 02:50:18.508642 [DBG 2] using ml_sequence
Nov 21 2024 02:50:18.510234 [DBG 2] Set polygons from request
Nov 21 2024 02:50:18.510314 [DBG 2] Found stream_sequence in mlapi config, ignoring objectconfig.ini
Nov 21 2024 02:50:18.510431 [DBG 1] Object Recognition requested
Nov 21 2024 02:50:18.510468 [DBG 1] Calling detect streams
Nov 21 2024 02:50:18.510526 [DBG 2] Called detect_stream(stream=368365, ml_overrides={'model_sequence': 'object', 'object': {'pattern': '(person|car|motorbike|bus|truck|boat)'}, 'face': {'pattern': '.*'}, 'alpr': {'pattern': '.*'}}, options={'frame_strategy': 'most_models', 'frame_set': 'snapshot,alarm', 'contig_frames_before_error': 5, 'max_attempts': 3, 'sleep_between_attempts': 4, 'resize': 800, 'convert_snapshot_to_fid': 'yes', 'api': <pyzm.api.ZMApi object at 0x7fbe8acd07c0>, 'polygons': [], 'mid': None})
Nov 21 2024 02:50:18.510563 [DBG 3] Using manual locking as we are only using one model
Nov 21 2024 02:50:18.510735 [DBG 2] Media get SSL certificate check has been disbled
Nov 21 2024 02:50:18.510809 [DBG 2] Using URL 368365 for stream
Nov 21 2024 02:50:18.510853 [DBG 2] We will only process frames: ['snapshot', 'alarm']
Nov 21 2024 02:50:18.510886 [DBG 2] No need to start streams, we are picking images from http://192.168.1.2:8080/zm/index.php?view=image&eid=368365
Nov 21 2024 02:50:18.510942 [DBG 2] in detect_stream() polygons=[]
Nov 21 2024 02:50:18.510981 [DBG 2] Trying to convert snapshot to a real frame id
Nov 21 2024 02:50:18.511033 [DBG 3] No need to relogin as access token still has 39.96499968333333 minutes remaining
Nov 21 2024 02:50:18.511084 [DBG 3] make_request called with url=http://192.168.1.2:8080/zm/api/events/368365.json payload={} type=get query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJab25lTWluZGVyIiwiaWF0IjoxNzMyMTUyNjE2LCJleHAiOjE3MzIxNTk4MTYsInVzZXIiOiJhZG1pbiIsInR5cGUiOiJhY2Nlc3MifQ._M8HAJMldeqq0iTrpwWZKob2UfK9vXX41cqRhRH-rfs'}
Nov 21 2024 02:50:18.647603 [DBG 2] At the point of analysis, Event:368365 snapshot frame id was:180,so using it
Nov 21 2024 02:50:18.647718 [DBG 3] Reading http://192.168.1.2:8080/zm/index.php?view=image&eid=368365&fid=180
Nov 21 2024 02:50:18.647793 [DBG 3] No need to relogin as access token still has 39.96272045 minutes remaining
Nov 21 2024 02:50:18.647866 [DBG 3] make_request called with url=http://192.168.1.2:8080/zm/index.php?view=image&eid=368365&fid=180 payload={} type=get query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJab25lTWluZGVyIiwiaWF0IjoxNzMyMTUyNjE2LCJleHAiOjE3MzIxNTk4MTYsInVzZXIiOiJhZG1pbiIsInR5cGUiOiJhY2Nlc3MifQ._M8HAJMldeqq0iTrpwWZKob2UfK9vXX41cqRhRH-rfs'}
Nov 21 2024 02:50:19.090256 [DBG 1] perf: Starting for frame:180
Nov 21 2024 02:50:19.090360 [DBG 1] Sequence of detection types to execute: ['object']
Nov 21 2024 02:50:19.090400 [DBG 1] ============ Frame: 180 Running object detection type in sequence ==================
Nov 21 2024 02:50:19.090438 [DBG 3] object has a same_model_sequence strategy of most_unique
Nov 21 2024 02:50:19.090472 [DBG 1] --------- Frame:180 Running variation: #1 -------------
Nov 21 2024 02:50:19.091267 [DBG 2] Waiting for pyzm_uid0_tpu_lock portalock...
Nov 21 2024 02:50:19.091638 [DBG 2] Got pyzm_uid0_tpu_lock portalock
Nov 21 2024 02:50:19.091685 [DBG 1] |---------- TPU (input image: 800w*300h) ----------|
Nov 21 2024 02:50:19.739106 [DBG 2] Released portalock pyzm_uid0_tpu_lock
Nov 21 2024 02:50:19.739235 [DBG 1] perf: processor:tpu Coral TPU detection took: 647.29 ms
Nov 21 2024 02:50:19.739292 [DBG 3] Coral object returning: [],[],[]
Nov 21 2024 02:50:19.739351 [DBG 2] core model detection over, got 0 objects. Now filtering
Nov 21 2024 02:50:19.739400 [DBG 2] Max object size found to be: 90%
Nov 21 2024 02:50:19.739485 [DBG 2] Converted 90% to 216000.0
Nov 21 2024 02:50:19.739539 [DBG 2] Returning filtered list of 0 objects.
Nov 21 2024 02:50:19.739588 [DBG 2] This model iteration inside object found: labels: [],conf:[]
Nov 21 2024 02:50:19.739641 [DBG 2] Called _filter_detections(seq=object, box=[], label=[], conf=[], polygons=[], h=300, w=800, model_names=[])
Nov 21 2024 02:50:19.739689 [DBG 2] Max object size found to be: 90%
Nov 21 2024 02:50:19.739753 [DBG 2] Converted 90% to 216000.0
Nov 21 2024 02:50:19.739867 [DBG 2] No polygons, adding full image polygon: {'name': 'full_image', 'value': [(0, 0), (1536, 0), (1536, 576), (0, 576)], 'pattern': None}
Nov 21 2024 02:50:19.739962 [DBG 3] resized polygons x=0.5208333333333334/y=0.5208333333333334: [{'name': 'full_image', 'value': [(0, 0), (800, 0), (800, 300), (0, 300)], 'pattern': None}]
Nov 21 2024 02:50:19.740023 [DBG 2] We did not find any object matches in frame: 180
Nov 21 2024 02:50:19.740081 [DBG 3] Reading http://192.168.1.2:8080/zm/index.php?view=image&eid=368365&fid=alarm
Nov 21 2024 02:50:19.740148 [DBG 3] No need to relogin as access token still has 39.94451445 minutes remaining
Nov 21 2024 02:50:19.740223 [DBG 3] make_request called with url=http://192.168.1.2:8080/zm/index.php?view=image&eid=368365&fid=alarm payload={} type=get query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJab25lTWluZGVyIiwiaWF0IjoxNzMyMTUyNjE2LCJleHAiOjE3MzIxNTk4MTYsInVzZXIiOiJhZG1pbiIsInR5cGUiOiJhY2Nlc3MifQ._M8HAJMldeqq0iTrpwWZKob2UfK9vXX41cqRhRH-rfs'}
Nov 21 2024 02:50:19.799326 [DBG 1] perf: Starting for frame:alarm
Nov 21 2024 02:50:19.799466 [DBG 1] Sequence of detection types to execute: ['object']
Nov 21 2024 02:50:19.799519 [DBG 1] ============ Frame: alarm Running object detection type in sequence ==================
Nov 21 2024 02:50:19.799569 [DBG 3] object has a same_model_sequence strategy of most_unique
Nov 21 2024 02:50:19.799616 [DBG 1] --------- Frame:alarm Running variation: #1 -------------
Nov 21 2024 02:50:19.800632 [DBG 2] Waiting for pyzm_uid0_tpu_lock portalock...
Nov 21 2024 02:50:19.801153 [DBG 2] Got pyzm_uid0_tpu_lock portalock
Nov 21 2024 02:50:19.801281 [DBG 1] |---------- TPU (input image: 800w*300h) ----------|
Nov 21 2024 02:50:20.405755 [DBG 2] Released portalock pyzm_uid0_tpu_lock
Nov 21 2024 02:50:20.405892 [DBG 1] perf: processor:tpu Coral TPU detection took: 604.32 ms
Nov 21 2024 02:50:20.405950 [DBG 3] Coral object returning: [],[],[]
Nov 21 2024 02:50:20.406009 [DBG 2] core model detection over, got 0 objects. Now filtering
Nov 21 2024 02:50:20.406057 [DBG 2] Max object size found to be: 90%
Nov 21 2024 02:50:20.406142 [DBG 2] Converted 90% to 216000.0
Nov 21 2024 02:50:20.406196 [DBG 2] Returning filtered list of 0 objects.
Nov 21 2024 02:50:20.406245 [DBG 2] This model iteration inside object found: labels: [],conf:[]
Nov 21 2024 02:50:20.406317 [DBG 2] Called _filter_detections(seq=object, box=[], label=[], conf=[], polygons=[{'name': 'full_image', 'value': [(0, 0), (800, 0), (800, 300), (0, 300)], 'pattern': None}], h=300, w=800, model_names=[])
Nov 21 2024 02:50:20.406401 [DBG 2] Max object size found to be: 90%
Nov 21 2024 02:50:20.406469 [DBG 2] Converted 90% to 216000.0
Nov 21 2024 02:50:20.406530 [DBG 2] We did not find any object matches in frame: alarm
Nov 21 2024 02:50:20.406587 [DBG 2] pyzm_uid0_tpu_lock portalock already released
Nov 21 2024 02:50:20.406650 [DBG 1] perf: TOTAL detection sequence (with image loads) took: 1896.04 ms  to process 368365
Nov 21 2024 02:50:20.406752 [DBG 1] Returning {'matched_data': {'boxes': [], 'error_boxes': [], 'labels': [], 'confidences': [], 'frame_id': None, 'model_names': [], 'image_dimensions': {'original': (576, 1536), 'resized': (300, 800)}, 'image': None, 'polygons': [{'name': 'full_image', 'value': [(0, 0), (800, 0), (800, 300), (0, 300)], 'pattern': None}]}, 'all_matches': []}
Nov 21 2024 02:52:22.058689 [DBG 1] Monitor ID not specified, or not found in mlapi config, using zm_detect overrides
Nov 21 2024 02:52:22.058783 [DBG 2] using ml_sequence
Nov 21 2024 02:52:22.060522 [DBG 2] Set polygons from request
Nov 21 2024 02:52:22.060605 [DBG 2] Found stream_sequence in mlapi config, ignoring objectconfig.ini
Nov 21 2024 02:52:22.060724 [DBG 1] Object Recognition requested
Nov 21 2024 02:52:22.060762 [DBG 1] Calling detect streams
Nov 21 2024 02:52:22.060819 [DBG 2] Called detect_stream(stream=368366, ml_overrides={'model_sequence': 'object', 'object': {'pattern': '(person|car|motorbike|bus|truck|boat)'}, 'face': {'pattern': '.*'}, 'alpr': {'pattern': '.*'}}, options={'frame_strategy': 'most_models', 'frame_set': 'snapshot,alarm', 'contig_frames_before_error': 5, 'max_attempts': 3, 'sleep_between_attempts': 4, 'resize': 800, 'convert_snapshot_to_fid': 'yes', 'api': <pyzm.api.ZMApi object at 0x7fbe8acd07c0>, 'polygons': [], 'mid': None})
Nov 21 2024 02:52:22.060856 [DBG 3] Using manual locking as we are only using one model
Nov 21 2024 02:52:22.061031 [DBG 2] Media get SSL certificate check has been disbled
Nov 21 2024 02:52:22.061126 [DBG 2] Using URL 368366 for stream
Nov 21 2024 02:52:22.061173 [DBG 2] We will only process frames: ['snapshot', 'alarm']
Nov 21 2024 02:52:22.061239 [DBG 2] No need to start streams, we are picking images from http://192.168.1.2:8080/zm/index.php?view=image&eid=368366
Nov 21 2024 02:52:22.061305 [DBG 2] in detect_stream() polygons=[]
Nov 21 2024 02:52:22.061347 [DBG 2] Trying to convert snapshot to a real frame id
Nov 21 2024 02:52:22.061412 [DBG 3] No need to relogin as access token still has 37.90582676666667 minutes remaining
Nov 21 2024 02:52:22.061471 [DBG 3] make_request called with url=http://192.168.1.2:8080/zm/api/events/368366.json payload={} type=get query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJab25lTWluZGVyIiwiaWF0IjoxNzMyMTUyNjE2LCJleHAiOjE3MzIxNTk4MTYsInVzZXIiOiJhZG1pbiIsInR5cGUiOiJhY2Nlc3MifQ._M8HAJMldeqq0iTrpwWZKob2UfK9vXX41cqRhRH-rfs'}
Nov 21 2024 02:52:22.193241 [DBG 2] At the point of analysis, Event:368366 snapshot frame id was:1525,so using it
Nov 21 2024 02:52:22.193360 [DBG 3] Reading http://192.168.1.2:8080/zm/index.php?view=image&eid=368366&fid=1525
Nov 21 2024 02:52:22.193436 [DBG 3] No need to relogin as access token still has 37.9036264 minutes remaining
Nov 21 2024 02:52:22.193512 [DBG 3] make_request called with url=http://192.168.1.2:8080/zm/index.php?view=image&eid=368366&fid=1525 payload={} type=get query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJab25lTWluZGVyIiwiaWF0IjoxNzMyMTUyNjE2LCJleHAiOjE3MzIxNTk4MTYsInVzZXIiOiJhZG1pbiIsInR5cGUiOiJhY2Nlc3MifQ._M8HAJMldeqq0iTrpwWZKob2UfK9vXX41cqRhRH-rfs'}
Nov 21 2024 02:52:22.486345 [DBG 1] HTTP error: 404 Client Error: Not Found for url: http://192.168.1.2:8080/zm/index.php?view=image&eid=368366&fid=1525&token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJab25lTWluZGVyIiwiaWF0IjoxNzMyMTUyNjE2LCJleHAiOjE3MzIxNTk4MTYsInVzZXIiOiJhZG1pbiIsInR5cGUiOiJhY2Nlc3MifQ._M8HAJMldeqq0iTrpwWZKob2UfK9vXX41cqRhRH-rfs
Nov 21 2024 02:52:22.486450 [DBG 3] Raising BAD_IMAGE ValueError for a 404
Nov 21 2024 02:52:22.486532 [DBG 2] Failed attempt:1 of 3
Nov 21 2024 02:52:22.486582 [DBG 2] Sleeping for 4 seconds before retry
Nov 21 2024 02:52:26.489343 [DBG 3] No need to relogin as access token still has 37.832028333333334 minutes remaining
Nov 21 2024 02:52:26.489504 [DBG 3] make_request called with url=http://192.168.1.2:8080/zm/index.php?view=image&eid=368366&fid=1525 payload={} type=get query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJab25lTWluZGVyIiwiaWF0IjoxNzMyMTUyNjE2LCJleHAiOjE3MzIxNTk4MTYsInVzZXIiOiJhZG1pbiIsInR5cGUiOiJhY2Nlc3MifQ._M8HAJMldeqq0iTrpwWZKob2UfK9vXX41cqRhRH-rfs'}
Nov 21 2024 02:52:26.879494 [DBG 1] perf: Starting for frame:1525
Nov 21 2024 02:52:26.879643 [DBG 1] Sequence of detection types to execute: ['object']
Nov 21 2024 02:52:26.879694 [DBG 1] ============ Frame: 1525 Running object detection type in sequence ==================
Nov 21 2024 02:52:26.879741 [DBG 3] object has a same_model_sequence strategy of most_unique
Nov 21 2024 02:52:26.879785 [DBG 1] --------- Frame:1525 Running variation: #1 -------------
Nov 21 2024 02:52:26.880631 [DBG 2] Waiting for pyzm_uid0_tpu_lock portalock...
Nov 21 2024 02:52:26.881025 [DBG 2] Got pyzm_uid0_tpu_lock portalock
Nov 21 2024 02:52:26.881082 [DBG 1] |---------- TPU (input image: 800w*300h) ----------|

I see some errors:

Nov 21 2024 02:52:22.486345 [DBG 1] HTTP error: 404 Client Error: Not Found for url: http://192.168.1.2:8080/zm/index.php?view=image&eid=368366&fid=1525&token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJab25lTWluZGVyIiwiaWF0IjoxNzMyMTUyNjE2LCJleHAiOjE3MzIxNTk4MTYsInVzZXIiOiJhZG1pbiIsInR5cGUiOiJhY2Nlc3MifQ._M8HAJMldeqq0iTrpwWZKob2UfK9vXX41cqRhRH-rfs
Nov 21 2024 02:52:22.486450 [DBG 3] Raising BAD_IMAGE ValueError for a 404
Nov 21 2024 02:52:22.486532 [DBG 2] Failed attempt:1 of 3
Nov 21 2024 02:52:22.486582 [DBG 2] Sleeping for 4 seconds before retry

Could it be the root cause?

Thanks.

@apsilva
Copy link

apsilva commented Nov 21, 2024

search in logs, i found that the first timeout event it happen before:

Nov 21 01:03:39 sala zm_detect.py[317733]: ERR [zmesdetect_m6] [Error during remote post: HTTPConnectionPool(host='192.168.1.2', port=5000): Read timed out. (read timeout=10)]

At same minute in mlapi.log, I see the same error, so it could be related..

Nov 21 2024 01:03:06.576141 [DBG 3] Raising BAD_IMAGE ValueError for a 404
Nov 21 2024 01:03:06.576202 [DBG 2] Failed attempt:1 of 3
Nov 21 2024 01:03:06.576250 [DBG 2] Sleeping for 4 seconds before retry

But here it looks OK....

Nov 21 2024 01:03:12.089441 [DBG 1] Returning {'matched_data': {'boxes': [], 'error_boxes': [], 'labels': [], 'confidences': [], 'frame_id': None, 'model_names': [], 'image_dimensions': {'original': (576, 1536), 'resized': (300, 800)}, 'image': None, 'polygons': [{'name': 'full_image', 'value': [(0, 0), (800, 0), (800, 300), (0, 300)], 'pattern': None}]}, 'all_matches': []}

i see the return result, but zm_detect.py is not receiving it..

Full minute log:

Nov 21 2024 01:03:06.146570 [DBG 1] Monitor ID not specified, or not found in mlapi config, using zm_detect overrides
Nov 21 2024 01:03:06.146677 [DBG 2] using ml_sequence
Nov 21 2024 01:03:06.148481 [DBG 2] Set polygons from request
Nov 21 2024 01:03:06.148578 [DBG 2] Found stream_sequence in mlapi config, ignoring objectconfig.ini
Nov 21 2024 01:03:06.148698 [DBG 1] Object Recognition requested
Nov 21 2024 01:03:06.148738 [DBG 1] Calling detect streams
Nov 21 2024 01:03:06.148798 [DBG 2] Called detect_stream(stream=368321, ml_overrides={'model_sequence': 'object', 'object': {'pattern': '(person|car|motorbike|bus|truck|boat)'}, 'face': {'pattern': '.*'}, 'alpr': {'pattern': '.*'}}, options={'frame_strategy': 'most_models', 'frame_set': 'snapshot,alarm', 'contig_frames_before_error': 5, 'max_attempts': 3, 'sleep_between_attempts': 4, 'resize': 800, 'convert_snapshot_to_fid': 'yes', 'api': <pyzm.api.ZMApi object at 0x7fbe8acd07c0>, 'polygons': [], 'mid': None})
Nov 21 2024 01:03:06.148844 [DBG 3] Using manual locking as we are only using one model
Nov 21 2024 01:03:06.149035 [DBG 2] Media get SSL certificate check has been disbled
Nov 21 2024 01:03:06.149125 [DBG 2] Using URL 368321 for stream
Nov 21 2024 01:03:06.149171 [DBG 2] We will only process frames: ['snapshot', 'alarm']
Nov 21 2024 01:03:06.149219 [DBG 2] No need to start streams, we are picking images from http://192.168.1.2:8080/zm/index.php?view=image&eid=368321
Nov 21 2024 01:03:06.149284 [DBG 2] in detect_stream() polygons=[]
Nov 21 2024 01:03:06.149324 [DBG 2] Trying to convert snapshot to a real frame id
Nov 21 2024 01:03:06.149381 [DBG 3] No need to relogin as access token still has 24.766878366666667 minutes remaining
Nov 21 2024 01:03:06.149436 [DBG 3] make_request called with url=http://192.168.1.2:8080/zm/api/events/368321.json payload={} type=get query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJab25lTWluZGVyIiwiaWF0IjoxNzMyMTQ1MjcyLCJleHAiOjE3MzIxNTI0NzIsInVzZXIiOiJhZG1pbiIsInR5cGUiOiJhY2Nlc3MifQ.pen_WNTJp1TYA4FsXdiqU6yUBcwMclpFQ_1Rpgvu2rY'}
Nov 21 2024 01:03:06.249535 [DBG 2] At the point of analysis, Event:368321 snapshot frame id was:3505,so using it
Nov 21 2024 01:03:06.249648 [DBG 3] Reading http://192.168.1.2:8080/zm/index.php?view=image&eid=368321&fid=3505
Nov 21 2024 01:03:06.249722 [DBG 3] No need to relogin as access token still has 24.76520605 minutes remaining
Nov 21 2024 01:03:06.249798 [DBG 3] make_request called with url=http://192.168.1.2:8080/zm/index.php?view=image&eid=368321&fid=3505 payload={} type=get query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJab25lTWluZGVyIiwiaWF0IjoxNzMyMTQ1MjcyLCJleHAiOjE3MzIxNTI0NzIsInVzZXIiOiJhZG1pbiIsInR5cGUiOiJhY2Nlc3MifQ.pen_WNTJp1TYA4FsXdiqU6yUBcwMclpFQ_1Rpgvu2rY'}
Nov 21 2024 01:03:06.576035 [DBG 1] HTTP error: 404 Client Error: Not Found for url: http://192.168.1.2:8080/zm/index.php?view=image&eid=368321&fid=3505&token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJab25lTWluZGVyIiwiaWF0IjoxNzMyMTQ1MjcyLCJleHAiOjE3MzIxNTI0NzIsInVzZXIiOiJhZG1pbiIsInR5cGUiOiJhY2Nlc3MifQ.pen_WNTJp1TYA4FsXdiqU6yUBcwMclpFQ_1Rpgvu2rY
Nov 21 2024 01:03:06.576141 [DBG 3] Raising BAD_IMAGE ValueError for a 404
Nov 21 2024 01:03:06.576202 [DBG 2] Failed attempt:1 of 3
Nov 21 2024 01:03:06.576250 [DBG 2] Sleeping for 4 seconds before retry
Nov 21 2024 01:03:10.577677 [DBG 3] No need to relogin as access token still has 24.693073933333334 minutes remaining
Nov 21 2024 01:03:10.577860 [DBG 3] make_request called with url=http://192.168.1.2:8080/zm/index.php?view=image&eid=368321&fid=3505 payload={} type=get query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJab25lTWluZGVyIiwiaWF0IjoxNzMyMTQ1MjcyLCJleHAiOjE3MzIxNTI0NzIsInVzZXIiOiJhZG1pbiIsInR5cGUiOiJhY2Nlc3MifQ.pen_WNTJp1TYA4FsXdiqU6yUBcwMclpFQ_1Rpgvu2rY'}
Nov 21 2024 01:03:10.942440 [DBG 1] perf: Starting for frame:3505
Nov 21 2024 01:03:10.942554 [DBG 1] Sequence of detection types to execute: ['object']
Nov 21 2024 01:03:10.942592 [DBG 1] ============ Frame: 3505 Running object detection type in sequence ==================
Nov 21 2024 01:03:10.942628 [DBG 3] object has a same_model_sequence strategy of most_unique
Nov 21 2024 01:03:10.942660 [DBG 1] --------- Frame:3505 Running variation: #1 -------------
Nov 21 2024 01:03:10.943396 [DBG 2] Waiting for pyzm_uid0_tpu_lock portalock...
Nov 21 2024 01:03:10.943755 [DBG 2] Got pyzm_uid0_tpu_lock portalock
Nov 21 2024 01:03:10.943799 [DBG 1] |---------- TPU (input image: 800w*300h) ----------|
Nov 21 2024 01:03:11.499573 [DBG 2] Released portalock pyzm_uid0_tpu_lock
Nov 21 2024 01:03:11.499707 [DBG 1] perf: processor:tpu Coral TPU detection took: 555.64 ms
Nov 21 2024 01:03:11.499763 [DBG 3] Coral object returning: [],[],[]
Nov 21 2024 01:03:11.499822 [DBG 2] core model detection over, got 0 objects. Now filtering
Nov 21 2024 01:03:11.499870 [DBG 2] Max object size found to be: 90%
Nov 21 2024 01:03:11.499952 [DBG 2] Converted 90% to 216000.0
Nov 21 2024 01:03:11.500005 [DBG 2] Returning filtered list of 0 objects.
Nov 21 2024 01:03:11.500053 [DBG 2] This model iteration inside object found: labels: [],conf:[]
Nov 21 2024 01:03:11.500105 [DBG 2] Called _filter_detections(seq=object, box=[], label=[], conf=[], polygons=[], h=300, w=800, model_names=[])
Nov 21 2024 01:03:11.500152 [DBG 2] Max object size found to be: 90%
Nov 21 2024 01:03:11.500213 [DBG 2] Converted 90% to 216000.0
Nov 21 2024 01:03:11.500285 [DBG 2] No polygons, adding full image polygon: {'name': 'full_image', 'value': [(0, 0), (1536, 0), (1536, 576), (0, 576)], 'pattern': None}
Nov 21 2024 01:03:11.500373 [DBG 3] resized polygons x=0.5208333333333334/y=0.5208333333333334: [{'name': 'full_image', 'value': [(0, 0), (800, 0), (800, 300), (0, 300)], 'pattern': None}]
Nov 21 2024 01:03:11.500433 [DBG 2] We did not find any object matches in frame: 3505
Nov 21 2024 01:03:11.500489 [DBG 3] Reading http://192.168.1.2:8080/zm/index.php?view=image&eid=368321&fid=alarm
Nov 21 2024 01:03:11.500555 [DBG 3] No need to relogin as access token still has 24.6776921 minutes remaining
Nov 21 2024 01:03:11.500630 [DBG 3] make_request called with url=http://192.168.1.2:8080/zm/index.php?view=image&eid=368321&fid=alarm payload={} type=get query={'token': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJab25lTWluZGVyIiwiaWF0IjoxNzMyMTQ1MjcyLCJleHAiOjE3MzIxNTI0NzIsInVzZXIiOiJhZG1pbiIsInR5cGUiOiJhY2Nlc3MifQ.pen_WNTJp1TYA4FsXdiqU6yUBcwMclpFQ_1Rpgvu2rY'}
Nov 21 2024 01:03:11.536760 [DBG 1] perf: Starting for frame:alarm
Nov 21 2024 01:03:11.536939 [DBG 1] Sequence of detection types to execute: ['object']
Nov 21 2024 01:03:11.537081 [DBG 1] ============ Frame: alarm Running object detection type in sequence ==================
Nov 21 2024 01:03:11.537132 [DBG 3] object has a same_model_sequence strategy of most_unique
Nov 21 2024 01:03:11.537174 [DBG 1] --------- Frame:alarm Running variation: #1 -------------
Nov 21 2024 01:03:11.539481 [DBG 2] Waiting for pyzm_uid0_tpu_lock portalock...
Nov 21 2024 01:03:11.540036 [DBG 2] Got pyzm_uid0_tpu_lock portalock
Nov 21 2024 01:03:11.540167 [DBG 1] |---------- TPU (input image: 800w*300h) ----------|
Nov 21 2024 01:03:12.088573 [DBG 2] Released portalock pyzm_uid0_tpu_lock
Nov 21 2024 01:03:12.088681 [DBG 1] perf: processor:tpu Coral TPU detection took: 548.27 ms
Nov 21 2024 01:03:12.088730 [DBG 3] Coral object returning: [],[],[]
Nov 21 2024 01:03:12.088782 [DBG 2] core model detection over, got 0 objects. Now filtering
Nov 21 2024 01:03:12.088824 [DBG 2] Max object size found to be: 90%
Nov 21 2024 01:03:12.088895 [DBG 2] Converted 90% to 216000.0
Nov 21 2024 01:03:12.088942 [DBG 2] Returning filtered list of 0 objects.
Nov 21 2024 01:03:12.088985 [DBG 2] This model iteration inside object found: labels: [],conf:[]
Nov 21 2024 01:03:12.089072 [DBG 2] Called _filter_detections(seq=object, box=[], label=[], conf=[], polygons=[{'name': 'full_image', 'value': [(0, 0), (800, 0), (800, 300), (0, 300)], 'pattern': None}], h=300, w=800, model_names=[])
Nov 21 2024 01:03:12.089118 [DBG 2] Max object size found to be: 90%
Nov 21 2024 01:03:12.089174 [DBG 2] Converted 90% to 216000.0
Nov 21 2024 01:03:12.089252 [DBG 2] We did not find any object matches in frame: alarm
Nov 21 2024 01:03:12.089303 [DBG 2] pyzm_uid0_tpu_lock portalock already released
Nov 21 2024 01:03:12.089354 [DBG 1] perf: TOTAL detection sequence (with image loads) took: 5940.46 ms  to process 368321
Nov 21 2024 01:03:12.089441 [DBG 1] Returning {'matched_data': {'boxes': [], 'error_boxes': [], 'labels': [], 'confidences': [], 'frame_id': None, 'model_names': [], 'image_dimensions': {'original': (576, 1536), 'resized': (300, 800)}, 'image': None, 'polygons': [{'name': 'full_image', 'value': [(0, 0), (800, 0), (800, 300), (0, 300)], 'pattern': None}]}, 'all_matches': []}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pliablepixels @apsilva @fennec622