CVO Azure：ノードのコントローラフェイルオーバーを実行できません：同期されていないログが繰り返されます

最後の更新
PDFとして保存

Views:: 6

Visibility:: Public

Votes:: 0

Category:: cloud-volumes-ontap-cvo

Specialty:: cloud<a>2009739296</a>

Last Updated:

環境

Cloud Volumes ONTAP（CVO）
Microsoft Azure
非同期ログ
入出力（IO）

問題

AutoSupport ems.log.gz ログで、同期されていないエラーメッセージが繰り返し報告され、次のようにリカバリされます。

Mon Oct 02 04:24:50 -0400 [cvo-01: cf_main: cf.fsm.takeoverOfPartnerDisabled:error]: Failover monitor: takeover of cvo-02 disabled (unsynchronized log). Mon Oct 02 04:24:53 -0400 [cvo-01: cf_main: cf.fsm.takeoverOfPartnerEnabled:notice]: Failover monitor: takeover of cvo-02 enabled

EMS.LOG.GZ レポートは、基盤となるQPSがpoll_timeoutエラーを取得しています。

Tue Sep 05 17:50:07 -0400 [cvo-02: mcc_cfd_rnic: nvmm.mirror.aborting:debug]: mirror of sysid 1, partner_type HA Partner and mirror state NVMM_MIRROR_ONLINE is aborted because of reason NVMM_ERR_POLL_TIMEOUT. Tue Sep 05 17:50:07 -0400 [cvo-02: mcc_cfd_rnic: mirror.stream.qp.error:debug]: params: {'mirror': 'HA Partner', 'qp_name': 'RAID', 'error': 'NVMM_ERR_POLL_TIMEOUT'} Tue Sep 05 17:50:07 -0400 [cvo-02: mcc_cfd_rnic: mirror.stream.qp.error:debug]: params: {'mirror': 'HA Partner', 'qp_name': 'MISC', 'error': 'NVMM_ERR_STREAM'} Tue Sep 05 17:50:07 -0400 [cvo-02: nvmm_error: mirror.stream.qp.error:debug]: params: {'mirror': 'HA Partner', 'qp_name': 'WAFL', 'error': 'NVMM_ERR_MIRROR_COMPLETION'} Tue Sep 05 17:50:07 -0400 [cvo-02: nvmm_error: mirror.stream.qp.error:debug]: params: {'mirror': 'HA Partner', 'qp_name': 'WAFL', 'error': 'NVMM_ERR_STREAM'} Tue Sep 05 17:50:07 -0400 [cvo-02: nvmm_error: ems.engine.suppressed:debug]: Event 'rdma.rlib.event.error' suppressed 2 times in last 2754 seconds. Tue Sep 05 17:50:07 -0400 [cvo-02: nvmm_error: rdma.rlib.event.error:debug]: QP wafl event error: client disconnect. Tue Sep 05 17:50:07 -0400 [cvo-02: nvmm_error: nvmm.mirror.offlined:debug]: params: {'mirror': 'HA_PARTNER'} Tue Sep 05 17:50:07 -0400 [cvo-02: rastrace_dump: rastrace.dump.saved:debug]: A RAS trace dump for module IC instance 0 was stored in /etc/log/rastrace/IC_0_20230905_17:50:07:464435.dmp. Tue Sep 05 17:50:07 -0400 [cvo-02: cf_main: cf.fsm.takeoverByPartnerDisabled:error]: Failover monitor: takeover of cvo-02 by cvo-01 disabled (unsynchronized log).

SKTRACE.GZ ログは、RAID IOの完了に時間がかかっていることを示しています。

2023-09-06T19:15:32Z 3435898944123351 [14:0] BSD_RLIB_WARN: rlib_update_comp_info: qp wafl wr_id 300000000000000000 num_of_req 2 op_type 1 op_size 256 took time 2321466 usecs pending_size:2184520

2023-09-06T19:15:32Z 3435898944131159 [14:0] BSD_RLIB_WARN: rlib_update_comp_info: qp wafl wr_id 310000000000000000 num_of_req 2 op_type 1 op_size 344 took time 2320541 usecs pending_size:2184176

2023-09-06T19:15:32Z 3435898944168065 [14:0] BSD_RLIB_WARN: rlib_update_comp_info: qp wafl wr_id 312000000000000000 num_of_req 2 op_type 1 op_size 4160 took time 2313630 usecs pending_size:2180016