Health checks failed on VR

[sbrueseke]

I this this kind of alerts from time to time in our UI Alerts:
Health checks failed: 1 failing checks on router r-555-VM / 9d2d3c6c-d1a7-4fd0-8143-7a94b7032eb6

In management.log I find:
2025-01-22 17:41:24,521 WARN [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-ee7be78a) (logid:4a205737) Health checks failed: 1 failing checks on router r-555-VM / 9d2d3c6c-d1a7-4fd0-8143-7a94b7032eb6. Checking failed health checks to see if router needs recreate

In diagnostic cloud.log I am able to see this:
2025-01-22 17:50:39,837 ERROR Not able to setup source-nat for a regular router yet

This router was started at 2025-01-22 17:39:11
2025-01-22 17:39:11,339 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-81:ctx-48ac6c20 job-20526/job-20527 ctx-8fd308b0) (logid:bde48037) VM instance {"id":555,"instanceName":"r-555-VM","type":"DomainRouter","uuid":"9d2d3c6c-d1a7-4fd0-8143-7a94b7032eb6"} state transited from [Starting] to [Running] with event [OperationSucceeded]. VM's original host: Host {"id":21,"name":"pc-kvm-8","type":"Routing","uuid":"2fbe41ca-73c9-43d4-a16a-1a44be2b1bd2"}, new host: Host {"id":21,"name":"pc-kvm-8","type":"Routing","uuid":"2fbe41ca-73c9-43d4-a16a-1a44be2b1bd2"}, host before state transition: Host {"id":21,"name":"pc-kvm-8","type":"Routing","uuid":"2fbe41ca-73c9-43d4-a16a-1a44be2b1bd2"}

And it looks like the router is working fine. Is this kind of alert expected short after a start or should we investigate this?

Regards,
Swen

[weizhouapache]

@sbrueseke
is it the VR of shared, isolated network or vpc ?

[sbrueseke]

It is an isolated network in an advanced zone.

[weizhouapache]

you can get the health check result by cmk

get routerhealthcheckresults routerid=xxx

[sbrueseke]

what exactly is the id of the router? Docu is not really clear about it. Is it the id in vm_instance table, the uuid, or something else?
I guess it is the id from the vm_instance table?

(proIO) 🐱 > get routerhealthcheckresults routerid=555
{
"routerhealthchecks": {
"healthchecks": [
{
"checkname": "connectivity.test",
"checktype": "basic",
"details": "Successfully connected to router",
"lastupdated": "2025-01-23T10:01:37+0100",
"success": true
},
{
"checkname": "cpu_usage_check.py",
"checktype": "basic",
"details": "CPU Usage within limits with current at 3.7%",
"lastupdated": "2025-01-23T10:00:01+0100",
"success": true
},
{
"checkname": "dhcp.service",
"checktype": "basic",
"details": "service is running",
"lastupdated": "2025-01-23T10:00:01+0100",
"success": true
},
{
"checkname": "dhcp_check.py",
"checktype": "advanced",
"details": "All 1 VMs are present in dhcphosts.txt",
"lastupdated": "2025-01-23T10:00:01+0100",
"success": true
},
{
"checkname": "disk_space_check.py",
"checktype": "basic",
"details": "Sufficient free space is 2976 MB",
"lastupdated": "2025-01-23T10:00:04+0100",
"success": true
},
{
"checkname": "dns_check.py",
"checktype": "advanced",
"details": "All 1 VMs are present in /etc/hosts",
"lastupdated": "2025-01-23T10:00:01+0100",
"success": true
},
{
"checkname": "filesystem.writable.test",
"checktype": "basic",
"details": "Successfully written to file system",
"lastupdated": "2025-01-23T10:01:37+0100",
"success": true
},
{
"checkname": "gateways_check.py",
"checktype": "basic",
"details": "All 2 gateways are reachable via ping",
"lastupdated": "2025-01-23T10:00:04+0100",
"success": true
},
{
"checkname": "haproxy_check.py",
"checktype": "advanced",
"details": "No data provided to check, skipping",
"lastupdated": "2025-01-23T10:00:01+0100",
"success": true
},
{
"checkname": "iptables_check.py",
"checktype": "advanced",
"details": "No portforwarding rules provided to check, skipping",
"lastupdated": "2025-01-23T10:00:01+0100",
"success": true
},
{
"checkname": "loadbalancing.service",
"checktype": "basic",
"details": "service is running",
"lastupdated": "2025-01-23T10:00:01+0100",
"success": true
},
{
"checkname": "memory_usage_check.py",
"checktype": "basic",
"details": "Memory Usage within limits with current at 52.5651%",
"lastupdated": "2025-01-23T10:00:12+0100",
"success": true
},
{
"checkname": "router_version_check.py",
"checktype": "basic",
"details": "Template and scripts version match successful",
"lastupdated": "2025-01-23T10:00:12+0100",
"success": true
},
{
"checkname": "ssh.service",
"checktype": "basic",
"details": "service is running",
"lastupdated": "2025-01-23T10:00:01+0100",
"success": true
},
{
"checkname": "webserver.service",
"checktype": "basic",
"details": "service is running",
"lastupdated": "2025-01-23T10:00:01+0100",
"success": true
}
],
"routerid": "9d2d3c6c-d1a7-4fd0-8143-7a94b7032eb6"
}
}

[weizhouapache]

what exactly is the id of the router? Docu is not really clear about it. Is it the id in vm_instance table, the uuid, or something else? I guess it is the id from the vm_instance table?

normally we use uuid
id also works, but it might be unsupported in future versions.

health check results look good now

[sbrueseke]

Any idea what triggered this alert in the first place?

[weizhouapache]

@sbrueseke
I have no idea for now.

can you share more mgmt/agent logs

[sbrueseke]

It looks like we do get this kind of alert everytime a VR is starting. We run 4.19.1.2 at the moment.