集群出现大量shutdown socket failed.shutdown: Transport endpoint is not connected id日志

[epolar]

Description

使用的 6 主集群压测，发现性能不是很稳定，从节点的 CPU 不会稳定在一个值附近，测试机和主节点相对比较稳定。

从节点:

主节点:

查看主节点看到有很多这样的日志:

I0626 14:17:10.784807 132962 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583242
W0626 14:17:10.786993 132945 network.cpp:580] setResponse _isEnded, id:235581835 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.789252 132955 network.cpp:580] setResponse _isEnded, id:235582703 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.790391 132956 network.cpp:580] setResponse _isEnded, id:235583507 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.790848 132951 network.cpp:580] setResponse _isEnded, id:235583587 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.794304 132963 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.794345 132963 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583417
W0626 14:17:10.799111 132963 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.799168 132963 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235582423
W0626 14:17:10.808280 132948 network.cpp:580] setResponse _isEnded, id:235583600 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.812263 132963 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.812304 132963 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583128
W0626 14:17:10.823552 132950 network.cpp:580] setResponse _isEnded, id:235583592 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.844386 132962 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.844444 132962 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583362
I0626 14:17:10.850581 132962 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583379
W0626 14:17:10.854308 132963 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.854353 132963 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583568
W0626 14:17:10.879565 132959 network.cpp:580] setResponse _isEnded, id:235583524 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.898736 132962 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.898797 132962 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583617
W0626 14:17:10.901582 132963 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.901645 132963 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583458
W0626 14:17:10.911201 132948 network.cpp:580] setResponse _isEnded, id:235582864 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.928401 132963 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.928464 132963 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583066
W0626 14:17:10.935529 132963 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.935585 132963 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583576
W0626 14:17:10.937903 132962 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.937949 132962 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583619
W0626 14:17:10.940001 132962 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.940040 132962 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235582764
W0626 14:17:10.959430 132963 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.959478 132963 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583620
W0626 14:17:10.965763 132962 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.965811 132962 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583022
W0626 14:17:10.969278 132952 network.cpp:580] setResponse _isEnded, id:235583080 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.969380 132952 network.cpp:580] setResponse _isEnded, id:235583431 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.969821 132952 network.cpp:580] setResponse _isEnded, id:235583213 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.977566 132949 network.cpp:580] setResponse _isEnded, id:235582823 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:11.038833 132963 network.cpp:1052] drainRspCallback:Broken pipe

同样的日志在从节点也有:

W0626 14:30:15.104251  9455 network.cpp:580] setResponse _isEnded, id:221840542 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.105410  9464 network.cpp:580] setResponse _isEnded, id:221840559 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.105499  9464 network.cpp:580] setResponse _isEnded, id:221840567 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.105628  9464 network.cpp:580] setResponse _isEnded, id:221840390 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.107861  9452 network.cpp:580] setResponse _isEnded, id:221840187 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.107964  9452 network.cpp:580] setResponse _isEnded, id:221836457 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.108062  9452 network.cpp:580] setResponse _isEnded, id:221840493 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.112968  9453 network.cpp:580] setResponse _isEnded, id:221840478 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.113885  9466 network.cpp:580] setResponse _isEnded, id:221840430 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.113974  9466 network.cpp:580] setResponse _isEnded, id:221840604 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.114216  9466 network.cpp:580] setResponse _isEnded, id:221840344 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.114296  9466 network.cpp:580] setResponse _isEnded, id:221837082 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.114401  9452 network.cpp:580] setResponse _isEnded, id:221840391 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.114943  9452 network.cpp:580] setResponse _isEnded, id:221840489 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.115025  9452 network.cpp:580] setResponse _isEnded, id:221836942 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.115100  9452 network.cpp:580] setResponse _isEnded, id:221840615 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.115185  9452 network.cpp:580] setResponse _isEnded, id:221834083 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.122390  9452 network.cpp:580] setResponse _isEnded, id:221831828 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.122516  9452 network.cpp:580] setResponse _isEnded, id:221840579 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.122613  9452 network.cpp:580] setResponse _isEnded, id:221837026 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.122516  9467 network.cpp:580] setResponse _isEnded, id:221840281 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.125306  9455 network.cpp:580] setResponse _isEnded, id:221836982 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.130019  9455 network.cpp:580] setResponse _isEnded, id:221832154 addr:remote_endpoint: Transport endpoint is not connected

Your Environment

• Machine Specifications: 8U16G * 6
• Tendis Version: 2.7.0-rocksdb-v8.5.3
• Tendis Configuration:

bind 0.0.0.0
port 51002
daemon on
requirepass pwd
masterauth pwd
cluster-enabled yes
loglevel notice
logdir /data/tendis/51002/log
dumpdir /data/tendis/51002/dump
dir /data/tendis/51002/db
pidfile ./home/tendisplus.pid
slowlog ./home/log/slowlog
rocks.cache_index_and_filter_blocks 1
rocks.compress_type lz4
slowlog-log-slower-than 10000000
rocks.blockcachemb 2458

netiothreadnum 2
executorThreadNum 16
#executorWorkPoolSize 3
#rocks.max_background_jobs 6

#allow-cross-slot true

checkkeytypeforsetcmd false
noexpire true

rocks.disable_wal true

[spihiker]

The same to me.

I0702 17:59:08.742622 23262 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:502303623
I0702 17:59:08.742643 23262 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:502303625
I0702 17:59:08.742765 23262 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:502303624
I0702 17:59:08.742867 23262 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:502303626
I0702 17:59:08.742960 23262 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:502303642
I0702 17:59:08.743026 23262 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:502303630
I0702 17:59:08.743160 23262 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:502303631
I0702 17:59:08.743299 23262 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:502303643
I0702 17:59:08.743341 23262 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:502303646

[hankji]

The same to me.