Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

consul agent stuck on proxycfg #22045

Open
woodliu opened this issue Jan 2, 2025 · 2 comments
Open

consul agent stuck on proxycfg #22045

woodliu opened this issue Jan 2, 2025 · 2 comments

Comments

@woodliu
Copy link

woodliu commented Jan 2, 2025 •
edited
Loading

Overview of the Issue

This is the issue from consul agent, here is the configuration(mocked)

{
    "acl": {
        "default_policy": "deny",
        "down_policy": "async-cache",
        "enable_token_persistence": true,
        "enabled": true,
        "tokens": {
            "agent": "xxxxx"
        }
    },
    "addresses": {
        "http": "127.0.0.1"
    },
    "advertise_addr": "x.x.x.x",
    "auto_encrypt": {
        "tls": true
    },
    "bind_addr": "0.0.0.0",
    "data_dir": "/var/lib/consul/",
    "datacenter": "we1-prd",
    "disable_keyring_file": true,
    "disable_remote_exec": true,
    "disable_update_check": true,
    "encrypt": "EIK2bcOqT1h856tXINseeUW/xxxx",
    "leave_on_terminate": false,
    "limits": {
        "http_max_conns_per_client": 1800,
        "rpc_max_conns_per_client": 3600
    },
    "log_level": "TRACE",
    "node_id": "x-x-x-x-x",
    "node_meta": {
        "kafka": "role",
        "resource_group": "prd-infrastructure-central"
    },
    "node_name": "prd-infrastructure-central4.local",
    "retry_join": [
        "prd-consulserver-1.local",
        "prd-consulserver-2.local",
        "prd-consulserver-3.local",
        "prd-consulserver-4.local",
        "prd-consulserver-5.local"
    ],
    "server": false,
    "telemetry": {
        "disable_hostname": true,
        "prometheus_retention_time": "5m"
    },
    "tls": {
        "defaults": {
            "ca_file": "/etc/consul.d/consul-agent-ca.pem",
            "verify_outgoing": true
        },
        "https": {
            "verify_incoming": false
        },
        "internal_rpc": {
            "verify_server_hostname": true
        }
    },
    "ui": false
}

Reproduction Steps

Re-install consul, when it starts, it works normally, but after some time(may be minutes or hours), it can't response to some command like /v1/agent/self consul debug

-->

Consul info for both Client and Server

The consul info command is blocked and i can't login the consul server

Here is the consul build info:

# consul version
Consul v1.18.2
Revision 9fc827ca
Build Date 2024-05-16T19:10:00Z

Operating system and Environment details

Operating system: AlmaLinux release 8.10 (Cerulean Leopard)

Connection: Consul-agent-->consul-server

Log Fragments

It prints on syncing proxy services from local state forever


Jan 02 05:21:12 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:12.666Z [INFO]  agent.client.serf.lan: serf: EventMemberFailed: aks-x-23686585-x 10.140.11.139
Jan 02 05:21:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:24.510Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:21:47 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:47.888Z [TRACE] agent.grpc.balancer: shuffling server order: target=consul://prd.00000000-1111-2222-aee1-be366e116526/server.prd
Jan 02 05:21:47 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:47.888Z [DEBUG] agent.grpc.balancer: switching server: target=consul://prd.00000000-1111-2222-aee1-be366e116526/server.prd from=prd-10.60.1.2:8300 to=prd-10.60.1.1:8300
Jan 02 05:21:47 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:47.888Z [TRACE] agent: [core][Channel #1 SubChannel #113] Subchannel created
Jan 02 05:21:47 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:47.888Z [TRACE] agent: [core][Channel #1] Channel Connectivity change to CONNECTING
Jan 02 05:21:47 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:47.888Z [TRACE] agent: [core][Channel #1 SubChannel #111] Subchannel Connectivity change to SHUTDOWN
Jan 02 05:21:47 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:47.888Z [TRACE] agent: [core][Channel #1 SubChannel #113] Subchannel Connectivity change to CONNECTING
Jan 02 05:21:47 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:47.888Z [TRACE] agent: [core][Channel #1 SubChannel #113] Subchannel picks a new address "prd-10.60.1.1:8300" to connect
Jan 02 05:21:47 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:47.888Z [TRACE] agent: [core][Channel #1 SubChannel #111] Subchannel deleted
Jan 02 05:21:47 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:47.889Z [TRACE] agent.grpc.balancer: sub-connection state changed: target=consul://prd.00000000-1111-2222-aee1-be366e116526/server.prd server=prd-10.60.1.1:8300 state=CONNECTING
Jan 02 05:21:47 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:47.890Z [TRACE] agent.tlsutil: OutgoingRPCConfig: version=7
Jan 02 05:21:47 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:47.893Z [TRACE] agent.tlsutil: OutgoingRPCWrapper: version=7
Jan 02 05:21:47 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:47.893Z [TRACE] agent.tlsutil: OutgoingRPCConfig: version=7
Jan 02 05:21:47 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:47.894Z [TRACE] agent: [core][Channel #1 SubChannel #113] Subchannel Connectivity change to READY
Jan 02 05:21:47 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:47.894Z [TRACE] agent.grpc.balancer: sub-connection state changed: target=consul://prd.00000000-1111-2222-aee1-be366e116526/server.prd server=prd-10.60.1.1:8300 state=READY
Jan 02 05:21:47 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:47.894Z [TRACE] agent: [core][Channel #1] Channel Connectivity change to READY
Jan 02 05:21:47 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:47.897Z [DEBUG] agent.router.manager: Rebalanced servers, new active server: number_of_servers=5 active_server="prd-infrastructure-consul-server-4.local (Addr: tcp/10.60.1.3:8300) (DC: we1-prd)"
Jan 02 05:21:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:21:54.510Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:22:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:22:24.511Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:22:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:22:54.512Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:23:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:23:24.512Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:23:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:23:54.513Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:24:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:24:24.513Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:24:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:24:54.514Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:25:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:25:24.514Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:25:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:25:54.515Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:26:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:26:24.515Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:26:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:26:54.516Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:27:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:27:24.517Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:27:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:27:54.517Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:28:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:28:24.518Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:28:30 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:28:30.241Z [TRACE] agent.tlsutil: OutgoingRPCWrapper: version=7
Jan 02 05:28:30 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:28:30.242Z [TRACE] agent.tlsutil: OutgoingRPCConfig: version=7
Jan 02 05:28:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:28:54.519Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:29:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:29:24.520Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:29:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:29:54.521Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:30:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:30:24.521Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:30:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:30:54.522Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:31:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:31:24.522Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:31:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:31:54.523Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:32:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:32:24.524Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:32:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:32:54.525Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:33:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:33:24.525Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:33:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:33:54.526Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:34:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:34:24.526Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:34:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:34:54.527Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:35:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:35:24.528Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:35:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:35:54.529Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:36:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:36:24.529Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:36:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:36:54.530Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:37:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:37:24.531Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:37:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:37:54.531Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:38:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:38:24.531Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:38:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:38:54.532Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:39:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:39:24.533Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:39:54 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:39:54.534Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
Jan 02 05:40:24 prd-infrastructure-central4 consul[199933]: 2025-01-02T05:40:24.535Z [TRACE] agent.proxycfg.agent-state: syncing proxy services from local state
@woodliu
Copy link
Author

woodliu commented Jan 2, 2025

Some command works: like /v1/agent/services

@woodliu
Copy link
Author

woodliu commented Jan 3, 2025 •
edited
Loading

When run lsof the process, it shows
image

@woodliu woodliu mentioned this issue Jan 6, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@woodliu