複数のファンでノードが停止しています
環境
- AFF A300 、 AFF A200 、 FAS8200 、 FAS2650 、 FAS2620 ( SP )
- AFF A220 、 AFF C190 、 FAS2750 、 FAS2720 ( BMC )
問題
- 複数のファンに障害が発生してコントローラがシャットダウンした場合。
env_mgr: monitor.chassisFan.stop:error]: Chassis fan contains at least one stopped fan: SysFan3 F1 (1260 RPM)
env_mgr: monitor.chassisFan.stop:error]: Chassis fan contains at least one stopped fan: SysFan3 F2 (1260 RPM)
env_mgr: monitor.chassisFanFail.xMinShutdown:EMERGENCY]: Multiple Chassis Fan failure: System will shut down in 2 minutes.
monitor: monitor.globalStatus.critical:EMERGENCY]: Multiple fans has failed: Sysfan3 F1, Sysfan3 F2.
statd: monitor.fan.failed:alert]: Multiple fans has failed: Sysfan3 F1, Sysfan3 F2.
statd: monitor.shutdown.emergency:EMERGENCY]: Emergency shutdown: Environmental Reason Shutdown (Multiple fans failed)
sysconfig -M
そのファンの出力:
Fri Jun 18 2021, 23:48:28 MSK !FAN3!021819020448!441-00025!A1!
Sat Jun 19 2021, 01:04:18 MSK !FAN3: Error reading FRU EEPROM
Sat Jun 19 2021, 07:05:27 MSK !FAN3!021819020448!441-00025!A1!
- EMS - ファン交換後も問題は維持されます。
env_mgr: monitor.chassisFan.removed:alert]: Chassis fan SysFan3 is removed
env_mgr: monitor.chassisFan.ok:notice]: Chassis fan Sysfan3 Present is ok.
env_mgr: monitor.chassisFan.ok:notice]: Chassis fan Sysfan3 F1 is ok.
env_mgr: monitor.chassisFan.ok:notice]: Chassis fan Sysfan3 F2 is ok.
env_mgr: monitor.chassisFan.stop:error]: Chassis fan contains at least one stopped fan: SysFan3 F1 (failed)
env_mgr: monitor.chassisFan.stop:error]: Chassis fan contains at least one stopped fan: SysFan3 F2 (failed)
env_mgr: monitor.chassisFanFail.xMinShutdown:EMERGENCY]: Multiple Chassis Fan failure: System will shut down in 2 minutes.
monitor: monitor.globalStatus.critical:EMERGENCY]: Multiple fans has failed: Sysfan3 F1, Sysfan3 F2.
statd: monitor.shutdown.emergency:EMERGENCY]: Emergency shutdown: Environmental Reason Shutdown (Multiple fans failed)
statd: monitor.shutdown.emergency:EMERGENCY]: Emergency shutdown: Environmental Reason Shutdown (Multiple fans failed)
shutdown_thread0: ha.localNodeShutDown:notice]: Shutdown of the local node has been initiated with inhibit_takeover set to FALSE.
cf_main: cf.fsm.takeoverOfPartnerDisabled:error]: Failover monitor: takeover of cluster2-01 disabled (local halt in progress).
shutdown_thread0: kern.shutdown:notice]: System shut down because : "Environmental Shutdown".
Wed Jun 23 20:46:54 MSK [toaster1: mgwd: callhome.hm.alert.critical:alert]: Call home for Health Monitor process cphm: CriticalFanFruFaultAlert[021604000131].
Thu Jun 24 11:00:00 MSK [toaster1: statd: monitor.fan.failed:alert]: Multiple fans has failed: Sysfan3 F1, Sysfan3 F2.
Thu Jun 24 11:18:36 MSK [toaster1: env_mgr: monitor.chassisFan.stop:error]: Chassis fan contains at least one stopped fan: SysFan3 F1 (failed)
Thu Jun 24 11:18:37 MSK [toaster1: env_mgr: monitor.chassisFan.stop:error]: Chassis fan contains at least one stopped fan: SysFan3 F2 (failed)
Thu Jun 24 11:19:06 MSK [toaster1: env_mgr: callhome.c.fan.fru.fault:error]: Call home for CHASSIS FAN FRU FAILED: SysFan3 F1
Thu Jun 24 11:19:06 MSK [toaster1: env_mgr: callhome.c.fan.fru.fault:error]: Call home for CHASSIS FAN FRU FAILED: SysFan3 F2
Thu Jun 24 11:00:00 MSK [toaster2: statd: monitor.fan.failed:alert]: Multiple fans has failed: Sysfan3 F1, Sysfan3 F2.
Thu Jun 24 11:53:52 MSK [toaster2: env_mgr: monitor.chassisFan.stop:error]: Chassis fan contains at least one stopped fan: SysFan3 F1 (failed)
Thu Jun 24 11:53:52 MSK [toaster2: env_mgr: monitor.chassisFan.stop:error]: Chassis fan contains at least one stopped fan: SysFan3 F2 (failed)
Oct 25 09:15:45 [toaster1:monitor.chassisFanFail.xMinShutdown:EMERGENCY]: Multiple Chassis Fan failure: System will shut down in 2 minutes.
Oct 25 09:16:00 [toaster1:monitor.globalStatus.critical:EMERGENCY]: Multiple fans has failed: Sysfan3 F2, Sysfan3 F1.
login: Oct 25 09:16:56 [toaster1:object.store.unavailable:EMERGENCY]: Unable to connect to the object store "ndas_rashidn12" from node d654735a-7cc4-11e9-9553-00a0985a07aa. Reason: Access denied.
Oct 25 09:18:16 [toaster:monitor.shutdown.emergency:EMERGENCY]: Emergency shutdown: Environmental Reason Shutdown (Multiple fans failed)