Agent in "connecting" state

[nlindblo]

I noticed that one of my Cloudstack hosts was in "Connecting" state, I have tried to restart the agent service a couple of time and I am seeing this in sequence recurring in the log. Any suggestion on what I should be troubleshooting ? I can't see anything obvious in the log

2025-01-23 03:44:03,531 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Process agent startup answer, agent id = 0
2025-01-23 03:44:03,532 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Set agent id 0
2025-01-23 03:44:03,568 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Startup Response Received: agent id = 0
2025-01-23 03:45:03,651 INFO [cloud.agent.Agent] (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
2025-01-23 03:45:15,314 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2025-01-23 03:45:15,318 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.19.0.1
2025-01-23 03:45:15,320 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2025-01-23 03:45:15,367 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage
2025-01-23 03:45:15,368 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm
2025-01-23 03:45:15,405 INFO [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2025-01-23 03:45:15,405 INFO [cloud.agent.AgentShell] (main:null) (logid:) Using default Java settings for IPv6 preference for agent connection
2025-01-23 03:45:15,482 INFO [cloud.agent.Agent] (main:null) (logid:) id is 0
2025-01-23 03:45:15,494 ERROR [kvm.resource.LibvirtComputingResource] (main:null) (logid:) uefi properties file not found due to: Unable to find file uefi.properties.
2025-01-23 03:45:15,558 INFO [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one
2025-01-23 03:45:15,764 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) IO uring driver for Qemu: disabled
2025-01-23 03:45:15,846 INFO [kvm.storage.KVMStoragePoolManager] (main:null) (logid:) adding storage adaptor for com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor
2025-01-23 03:45:15,848 INFO [kvm.storage.KVMStoragePoolManager] (main:null) (logid:) adding storage adaptor for com.cloud.hypervisor.kvm.storage.StorPoolStorageAdaptor
2025-01-23 03:45:15,848 WARN [kvm.storage.KVMStoragePoolManager] (main:null) (logid:) Duplicate StorageAdaptor type PowerFlex, not loading com.cloud.hypervisor.kvm.storage.ScaleIOStorageAdaptor
2025-01-23 03:45:15,849 INFO [kvm.storage.KVMStoragePoolManager] (main:null) (logid:) adding storage adaptor for com.cloud.hypervisor.kvm.storage.IscsiAdmStorageAdaptor
2025-01-23 03:45:15,849 INFO [kvm.storage.KVMStoragePoolManager] (main:null) (logid:) adding storage adaptor for com.cloud.hypervisor.kvm.storage.FiberChannelAdapter
2025-01-23 03:45:15,855 INFO [kvm.storage.MultipathSCSIAdapterBase] (main:null) (logid:) Loaded FiberChannelAdapter for StorageLayer
2025-01-23 03:45:15,856 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) No libvirt.vif.driver specified. Defaults to BridgeVifDriver.
2025-01-23 03:45:16,007 INFO [cloud.serializer.GsonHelper] (main:null) (logid:) Default Builder inited.
2025-01-23 03:45:16,008 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) iscsi session clean up is disabled
2025-01-23 03:45:16,010 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) The [vm.memballoon.stats.period] property is set to '0', this prevents memory statistics from being displayed correctly. Adjust (increase) the value of this parameter to correct this.
2025-01-23 03:45:16,148 INFO [cloud.agent.Agent] (main:null) (logid:) Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 : host = 192.168.10.221 : port = 8250
2025-01-23 03:45:16,159 INFO [utils.nio.NioClient] (main:null) (logid:) Connecting to 192.168.10.221:8250
2025-01-23 03:45:16,164 INFO [utils.nio.Link] (main:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2025-01-23 03:45:16,688 INFO [utils.nio.NioClient] (main:null) (logid:) SSL: Handshake done
2025-01-23 03:45:16,690 INFO [utils.nio.NioClient] (main:null) (logid:) Connected to 192.168.10.221:8250
2025-01-23 03:45:16,757 INFO [utils.linux.KVMHostInfo] (Agent-Handler-1:null) (logid:) Fetching CPU speed from command "lscpu".
2025-01-23 03:45:16,792 INFO [utils.linux.KVMHostInfo] (Agent-Handler-1:null) (logid:) Command [lscpu | grep -i 'Model name' | head -n 1 | egrep -o '[[:digit:]].[[:digit:]]+GHz' | sed 's/GHz//g'] resulted in the value [3200] for CPU speed.
2025-01-23 03:45:16,840 INFO [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:) Host uses control group [tmpfs].
2025-01-23 03:45:16,841 INFO [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:) Setting host CPU max capacity to 0, as it uses cgroup v1.
2025-01-23 03:45:17,004 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion
2025-01-23 03:45:17,060 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool 330e58d9-1f0d-48e1-82ac-e7aae6414e62 (Filesystem) in libvirt
2025-01-23 03:45:17,066 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Found existing defined storage pool 330e58d9-1f0d-48e1-82ac-e7aae6414e62, using it.
2025-01-23 03:45:17,067 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Trying to fetch storage pool 330e58d9-1f0d-48e1-82ac-e7aae6414e62 from libvirt
2025-01-23 03:45:17,131 ERROR [kvm.resource.LibvirtComputingResource] (Thread-4:null) (logid:) LibvirtException was thrown in event loop:
org.libvirt.LibvirtException: An error occurred, but the cause is unknown
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Library.processEvent(Unknown Source)
at org.libvirt.Library.runEventLoop(Unknown Source)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.lambda$setupLibvirtEventListener$0(LibvirtComputingResource.java:3699)
at java.base/java.lang.Thread.run(Thread.java:829)
2025-01-23 03:45:17,141 ERROR [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) (logid:) Failed to get libvirt connection for domain event lifecycle
org.libvirt.LibvirtException: internal error: could not initialize domain event timer
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.domainEventRegister(Unknown Source)
at org.libvirt.Connect.domainEventRegister(Unknown Source)
at org.libvirt.Connect.addLifecycleListener(Unknown Source)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.setupLibvirtEventListener(LibvirtComputingResource.java:3712)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createLocalStoragePool(LibvirtComputingResource.java:3692)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.initialize(LibvirtComputingResource.java:3636)
at com.cloud.agent.Agent.sendStartup(Agent.java:460)
at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1125)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2025-01-23 03:45:17,272 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Process agent startup answer, agent id = 0
2025-01-23 03:45:17,274 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Set agent id 0
2025-01-23 03:45:17,301 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Startup Response Received: agent id = 0

[sbrueseke]

I see some errors in the logs. Which version of the agent is running?

[weizhouapache]

I see some errors in the logs. Which version of the agent is running?

and libvirt version as well ?

restarting libvirt/libvirtd might help

[nlindblo]

I restarted libvirtd and it seem to have recovered after that and the host is showing as up again.

Thanks for the support, I will monitor the situation to see if it happens again