S5700因收到大(dà)量STP TC報文導緻CPU沖高(gāo)
2014/11/24 15:14:06點擊:
問題描述
某用戶反饋其企業網絡中,其中一台S5700交換機,CPU異常,經常達到90%以上(shàng)。
組網拓撲(故障設備為ZZXS57SW5F01-A4):
組網拓撲(故障設備為ZZXS57SW5F01-A4):
告警信息
查詢設備CPU信息,設備最近曾出現CPU沖高(gāo)記錄
=====================================================
===============display cpu-usage===============
=====================================================
CPU Usage Stat. Cycle: 60 (Second)
CPU Usage : 18% Max: 97%
CPU Usage Stat. Time : 2014-10-07 11:19:29
CPU utilization for five seconds: 18%: one minute: 18%: five minutes: 18%
Max CPU Usage Stat. Time : 2014-09-11 16:37:54.
查詢設備日志(zhì)有(yǒu)大(dà)量TC報文日志(zhì)産生(shēng):
Oct 7 2014 11:06:20-05:13 ZZXS57SW5F01-A4 %%01INFO/4/SUPPRESS_LOG(l)[15]:Last message repeated 1 times.(InfoID=1092489232, ModuleName=MSTP, InfoAlias=RECEIVE_MSTITC)
Oct 7 2014 11:05:19-05:13 ZZXS57SW5F01-A4 %%01INFO/4/SUPPRESS_LOG(l)[16]:Last message repeated 3 times.(InfoID=1092489232, ModuleName=MSTP, InfoAlias=RECEIVE_MSTITC)
Oct 7 2014 11:04:12-05:13 ZZXS57SW5F01-A4 %%01INFO/4/SUPPRESS_LOG(l)[17]:Last message repeated 3 times.(InfoID=1092489232, ModuleName=MSTP, InfoAlias=RECEIVE_MSTITC)
=====================================================
===============display cpu-usage===============
=====================================================
CPU Usage Stat. Cycle: 60 (Second)
CPU Usage : 18% Max: 97%
CPU Usage Stat. Time : 2014-10-07 11:19:29
CPU utilization for five seconds: 18%: one minute: 18%: five minutes: 18%
Max CPU Usage Stat. Time : 2014-09-11 16:37:54.
查詢設備日志(zhì)有(yǒu)大(dà)量TC報文日志(zhì)産生(shēng):
Oct 7 2014 11:06:20-05:13 ZZXS57SW5F01-A4 %%01INFO/4/SUPPRESS_LOG(l)[15]:Last message repeated 1 times.(InfoID=1092489232, ModuleName=MSTP, InfoAlias=RECEIVE_MSTITC)
Oct 7 2014 11:05:19-05:13 ZZXS57SW5F01-A4 %%01INFO/4/SUPPRESS_LOG(l)[16]:Last message repeated 3 times.(InfoID=1092489232, ModuleName=MSTP, InfoAlias=RECEIVE_MSTITC)
Oct 7 2014 11:04:12-05:13 ZZXS57SW5F01-A4 %%01INFO/4/SUPPRESS_LOG(l)[17]:Last message repeated 3 times.(InfoID=1092489232, ModuleName=MSTP, InfoAlias=RECEIVE_MSTITC)
處理(lǐ)過程
1、因未在故障時(shí)采集信息,無法知道(dào)是具體(tǐ)那(nà)些(xiē)進程沖高(gāo),懷疑為大(dà)量TC報文導緻設備FTS類進程沖高(gāo),設備一直産生(shēng)TC報文日志(zhì),首先确定此TC報文是本設備産生(shēng)的,還(hái)是從其它設備收到的。使用display stp tc-bpdu statistics命令查詢此TC報文是否是在本設備端産生(shēng),還(hái)是其它設備收到的。經查詢該設備與核心B互連端口TC報文入方向一直增漲,且同時(shí)轉發至其它接入層交換機。
[ZZXS57SW5F01-A4]display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port TC(Send/Receive) TCN(Send/Receive)
0 GigabitEthernet0/0/51 29272/63 0/0
0 GigabitEthernet0/0/52 3/18363 0/0
2、可(kě)以判定此TC報文不是本設備産生(shēng)的,使用相同命令逐層排查TC報文入方向設備。确認此TC報文是在網絡中的那(nà)一台設備上(shàng)産生(shēng)的。
核心B:
<ZZXS57SW0B-C2>display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port TC(Send/Receive) TCN(Send/Receive)
0 GigabitEthernet0/0/1 12495/13 0/0
0 GigabitEthernet0/0/2 135/8349 0/0
0 GigabitEthernet0/0/3 13430/19 0/0
0 GigabitEthernet0/0/4 13784/14 0/0
0 GigabitEthernet0/0/5 14200/17 0/0
0 GigabitEthernet0/0/6 14687/10 0/0
0 GigabitEthernet0/0/14 14164/16 0/0
0 GigabitEthernet0/0/15 14164/16 0/0
0 GigabitEthernet0/0/16 14625/12 0/0
0 Eth-Trunk1 11012/4 0/0
核心A:
<ZZXS57SW0A-C2>display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port TC(Send/Receive) TCN(Send/Receive)
0 GigabitEthernet0/0/1 16754/7 0/0
0 GigabitEthernet0/0/2 17112/1 0/0
0 GigabitEthernet0/0/3 17462/11 0/0
0 GigabitEthernet0/0/4 17793/4 0/0
0 GigabitEthernet0/0/5 18118/5 0/0
0 GigabitEthernet0/0/6 18415/3 0/0
0 GigabitEthernet0/0/14 17791/3 0/0
0 GigabitEthernet0/0/15 18113/6 0/0
0 GigabitEthernet0/0/16 18435/4 0/0
0 Eth-Trunk1 4/11010 0/0
接入ZZXS57SW5F02-A4
<ZZXS57SW5F02-A4>display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port TC(Send/Receive) TCN(Send/Receive)
0 GigabitEthernet0/0/51 63/29347 0/0
0 GigabitEthernet0/0/52 15/14712 0/0
3、當查詢到其中一台接入設備時(shí),發現其TC報文隻有(yǒu)在出方向上(shàng)不斷有(yǒu)增漲計(jì)數(shù),有(yǒu)此可(kě)見該TC報文為此設備産生(shēng),可(kě)使用display stp topology-change 查詢該TC報文的信息。可(kě)以查詢到該設備GigabitEthernet0/0/51端口不斷由阻塞變為放開(kāi)後,由于狀态變為DETECTED而觸發拓撲變化。
[ZZXS57SW5F02-A4]display stp topology-change
CIST topology change information
Number of topology changes :8561
Time since last topology change :0 days 0h:0m:19s
Topology change initiator(detected) :GigabitEthernet0/0/51
Topology change last received from :3400-a3e0-379c
Number of generated topologychange traps : 2615
Number of suppressed topologychange traps: 71
4、使用命令display interface brief查詢該接入設備端口信息,發現該設備GigabitEthernet0/0/51端口入方向有(yǒu)大(dà)量錯包,說明(míng)此接口入方向光纖線纜有(yǒu)問題,排查線纜故障後問題解決。
GigabitEthernet0/0/51 up up 0.01% 0.02% 38068638 0
根因
[ZZXS57SW5F01-A4]display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port TC(Send/Receive) TCN(Send/Receive)
0 GigabitEthernet0/0/51 29272/63 0/0
0 GigabitEthernet0/0/52 3/18363 0/0
2、可(kě)以判定此TC報文不是本設備産生(shēng)的,使用相同命令逐層排查TC報文入方向設備。确認此TC報文是在網絡中的那(nà)一台設備上(shàng)産生(shēng)的。
核心B:
<ZZXS57SW0B-C2>display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port TC(Send/Receive) TCN(Send/Receive)
0 GigabitEthernet0/0/1 12495/13 0/0
0 GigabitEthernet0/0/2 135/8349 0/0
0 GigabitEthernet0/0/3 13430/19 0/0
0 GigabitEthernet0/0/4 13784/14 0/0
0 GigabitEthernet0/0/5 14200/17 0/0
0 GigabitEthernet0/0/6 14687/10 0/0
0 GigabitEthernet0/0/14 14164/16 0/0
0 GigabitEthernet0/0/15 14164/16 0/0
0 GigabitEthernet0/0/16 14625/12 0/0
0 Eth-Trunk1 11012/4 0/0
核心A:
<ZZXS57SW0A-C2>display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port TC(Send/Receive) TCN(Send/Receive)
0 GigabitEthernet0/0/1 16754/7 0/0
0 GigabitEthernet0/0/2 17112/1 0/0
0 GigabitEthernet0/0/3 17462/11 0/0
0 GigabitEthernet0/0/4 17793/4 0/0
0 GigabitEthernet0/0/5 18118/5 0/0
0 GigabitEthernet0/0/6 18415/3 0/0
0 GigabitEthernet0/0/14 17791/3 0/0
0 GigabitEthernet0/0/15 18113/6 0/0
0 GigabitEthernet0/0/16 18435/4 0/0
0 Eth-Trunk1 4/11010 0/0
接入ZZXS57SW5F02-A4
<ZZXS57SW5F02-A4>display stp tc-bpdu statistics
-------------------------- STP TC/TCN information --------------------------
MSTID Port TC(Send/Receive) TCN(Send/Receive)
0 GigabitEthernet0/0/51 63/29347 0/0
0 GigabitEthernet0/0/52 15/14712 0/0
3、當查詢到其中一台接入設備時(shí),發現其TC報文隻有(yǒu)在出方向上(shàng)不斷有(yǒu)增漲計(jì)數(shù),有(yǒu)此可(kě)見該TC報文為此設備産生(shēng),可(kě)使用display stp topology-change 查詢該TC報文的信息。可(kě)以查詢到該設備GigabitEthernet0/0/51端口不斷由阻塞變為放開(kāi)後,由于狀态變為DETECTED而觸發拓撲變化。
[ZZXS57SW5F02-A4]display stp topology-change
CIST topology change information
Number of topology changes :8561
Time since last topology change :0 days 0h:0m:19s
Topology change initiator(detected) :GigabitEthernet0/0/51
Topology change last received from :3400-a3e0-379c
Number of generated topologychange traps : 2615
Number of suppressed topologychange traps: 71
4、使用命令display interface brief查詢該接入設備端口信息,發現該設備GigabitEthernet0/0/51端口入方向有(yǒu)大(dà)量錯包,說明(míng)此接口入方向光纖線纜有(yǒu)問題,排查線纜故障後問題解決。
GigabitEthernet0/0/51 up up 0.01% 0.02% 38068638 0
根因
STP組網中産與STP計(jì)算(suàn)的設備互連端口因鏈路質量不好,導緻設備STP頻繁收斂,産生(shēng)大(dà)量TC報文,導緻收到此TC報文的設備部分CPU進程沖商,影(yǐng)響業務正常運行(xíng)。
建議與總結
全局配置stp tc-protection
配置後可(kě)以保證設備頻繁收到TC報文時(shí),每2秒(miǎo)周期內(nèi)最多(duō)隻處理(lǐ)1次表項刷新。從而減少(shǎo)MAC、ARP表項頻繁刷新對設備的造成的負擔。
配置後可(kě)以保證設備頻繁收到TC報文時(shí),每2秒(miǎo)周期內(nèi)最多(duō)隻處理(lǐ)1次表項刷新。從而減少(shǎo)MAC、ARP表項頻繁刷新對設備的造成的負擔。
- 上(shàng)一篇:S5700交換機配置SNMP團體(tǐ)名字符串長限制(zhì)? 2014/11/24
- 下一篇:AR2200雙鏈路出口配置VRRP與NQA聯動切換不生(shēng)效 2014/11/24