ONTAP Select でディスクが障害状態としてマークされている
環境
- ONTAP Select
- VMware ESXi 6.5 または VMware ESXi 6.7 を実行している HPE サーバ。
問題
Disk NET-1.1
障害:アグリゲートがオフラインで、データにアクセスできません。ログにも同様のエラーが表示されます。
ONTAP Select sysconfig -a ログ:
slot 0: Virtual SAS Host Adapter 0b
Firmware rev: 4.2.0
2 : NETAPP PHA-DISK 0001 69.6GB 512B/sect (0-2sVvOXocGKsPNbJvg2)
3 : NETAPP PHA-DISK 0001 11534.3GB 512B/sect (0-2sVvOXMVRBTAOO/Qc3) (Failed)
ONTAP Select EMS ログ:
Tue Jun 01 01:34:32 +0200 [Node01: pha_main000: mlm.excessive.TPlatency:notice]: Average latency of 276855us on target port 53059d50444f5476, has exceeded 250000us
Tue Jun 01 01:34:32 +0200 [Node01: wafl_exempt01: wafl.cp.toolong:error]: Aggregate aggr0 experienced a long CP.
Tue Jun 01 01:34:34 +0200 [Node01: pha_main000: shm.threshold.agrsvIOCount:notice]: Disk 0b.3 has exceeded 12 IOs which have latencies greater than the threshold.
Tue Jun 01 01:34:34 +0200 [Node01: wafl_exempt02: wafl.cp.toolong:error]: Aggregate Aggr1 experienced a long CP.
Tue Jun 01 01:35:00 +0200 [Node01: intr: ems.engine.suppressed:debug]: Event 'dev.driver.throttling' suppressed 4 times in last 602 seconds.
Tue Jun 01 01:35:00 +0200 [Node01: intr: dev.driver.throttling:notice]: mpt driver unit 3 throttling I/O requests due to long latency.
Tue Jun 01 01:39:39 +0200 [Node01: cam: cam.timeout.retry:notice]: CAM device driver I/O timeout. Details: CAM command timeout (retrying), Device da3 (4 retries left). Device ID: 0-2sVvOXMVRBTAOO/Qc3. Command: WRITE(16). CDB: 8a 00 00 00 00 02 bf 93 ef 08 00 00 01 00 00 00 - outstanding for 60016006 milliseconds.
Tue Jun 01 01:39:39 +0200 [Node01: cam: cam.timeout.retry:notice]: CAM device driver I/O timeout. Details: CAM command timeout (retrying), Device da3 (4 retries left). Device ID: 0-2sVvOXMVRBTAOO/Qc3. Command: WRITE(16). CDB: 8a 00 00 00 00 02 bf 93 ee 08 00 00 01 00 00 00 - outstanding for 60017276 milliseconds.
Tue Jun 01 01:39:39 +0200 [Node01: cam: cam.timeout.retry:notice]: CAM device driver I/O timeout. Details: CAM command timeout (retrying), Device da3 (4 retries left). Device ID: 0-2sVvOXMVRBTAOO/Qc3. Command: WRITE(16). CDB: 8a 00 00 00 00 02 bf 93 ed 08 00 00 01 00 00 00 - outstanding for 60017444 milliseconds.
Tue Jun 01 01:39:39 +0200 [Node01: cam: cam.timeout.retry:notice]: CAM device driver I/O timeout. Details: CAM command timeout (retrying), Device da3 (4 retries left). Device ID: 0-2sVvOXMVRBTAOO/Qc3. Command: WRITE(16). CDB: 8a 00 00 00 00 02 bf 93 ec 08 00 00 01 00 00 00 - outstanding for 60017585 milliseconds.
VMware VMkernel ログ:
2021-05-31T22:53:22.005Z cpu29:7799666)Admission failure in path: host/vim/vmvisor/plugins/smx:sfcb-ProviderMa.7799660:uw.7799660
2021-05-31T23:02:09.083Z cpu3:7800957)Admission failure in path: host/vim/vmvisor/plugins/smx:sfcb-ProviderMa.7800955:uw.7800955