ノードがパニック状態になり、リブート時にクラスタに再参加しなくなる
環境
- FAS2750 / AFF-A220
- AFF-A250
- MCC-IP（MetroCluster IP）
- MetroClusterスイッチRCFアップグレード
問題
- MCC-IPのスイッチへのRCFアップグレード中に、1台のコントローラでCLAM パニックが発生します
Aug 01 17:40:16 [node01:vifmgr.clus.linkdown:EMERGENCY]: The cluster port e0a on node Node-01 has gone down unexpectedly. Aug 01 17:47:34 [node01:vifmgr.clus.linkdown:EMERGENCY]: The cluster port e0a on node Node-01 has gone down unexpectedly. Aug 01 17:52:10 [node01:vifmgr.clus.linkdown:EMERGENCY]: The cluster port e0a on node Node-01 has gone down unexpectedly. Aug 01 18:21:34 [node01:vifmgr.clus.linkdown:EMERGENCY]: The cluster port e0b on node Node-01 has gone down unexpectedly. PANIC : Received PANIC packet from partner, receiving message is (Coredump and takeover initiated because Connectivity, Liveliness and Availability Monitor (CLAM) has determined this node is out of quorum.)
- リブート時にクラスタポートe0a/e0bは稼働しているが、ノードは正常ではない
::*> network port show -role cluster
Auto-Negot Duplex Speed (Mbps)
Node Port Role Link MTU Admin/Oper Admin/Oper Admin/Oper
------ ------ ------------ ---- ----- ----------- ---------- ------------
node01
e0a cluster up 9000 true/true full/full auto/10000
e0b cluster up 9000 true/true full/full auto/10000
::> cluster show
Node Health Eligibility Epsilon
-------------------- ------- ------------ ------------
node01 false true false
node02 false true false
storage failover showノードがアプリケーションを開始していないことを報告します
::> storage failover show
Takeover
Node Partner Possible State Description
-------------- -------------- -------- -------------------------------------
node01
node02 true Connected to node02
node02
node01 true Connected to node01.
Waiting for cluster applications to
come online on the local node.
Offline applications: vldb, vifmgr,
bcomd, crs, scsi blade, clam.
2 entries were displayed.