ピアクラスタを使用できないため、FlexGroup のsnapmirror initializeが失敗します
環境
- ONTAP 9
- FlexGroup
- XDP
- クラスタ ピアリング
問題
- FlexGroup のSnapMirrorの初期化に失敗しました
- デスティネーションエラーの収集:
Failed to complete update operation on one or more item relationships
Volume <dr_vserver:dr_volume> is not initialized
Failed to complete update operation on one or more item relationships
Relationship with destination <dr_vserver:dr_volume> not found
Peer cluster <cluster_name> is no longer available
SnapMirror-audit-log:
Wed Sep 28 11:10:36 2022 FlexGroupInitialize[Sep 28 11:10:36]:c7b4e440-2d28-xxxx-xxxx-xxxxxxxxxxxx Operation-Uuid=bd7135bc-3ed2-11ed-b811-d039ea995e93 Group=flexgroup Operation-Cookie=0 action=Start source=svm1:vol1 destination=dr_svm1:dr_vol1
Wed Sep 28 11:10:39 2022 Initialize[Sep 28 11:10:39]:c7ae923e-2d28-xxxx-xxxx-xxxxxxxxxxxx Operation-Uuid=75dcc6b4-1f2c-4830-a878-7daf2cf240d0 Group=flexgroup Operation-Cookie=0 action=Start source=svm1:vol1__0001 destination=dr_svm1:dr_vol1__0001
Wed Sep 28 11:10:39 2022 Initialize[Sep 28 11:10:39]:c7ae928e-2d28-xxxx-xxxx-xxxxxxxxxxxx Operation-Uuid=6b99cdd2-3e2a-490c-966c-826edfb01a06 Group=flexgroup Operation-Cookie=0 action=Start source=svm1:vol1__0003 destination=dr_svm1:dr_vol1__0003
Wed Sep 28 11:10:39 2022 Initialize[Sep 28 11:10:39]:c7ae92cc-2d28-xxxx-xxxx-xxxxxxxxxxxx Operation-Uuid=56292615-abbe-4845-8029-0fd3c5bc518f Group=flexgroup Operation-Cookie=0 action=Start source=svm1:vol1__0005 destination=dr_svm1:dr_vol1__0005
Wed Sep 28 11:10:39 2022 Initialize[Sep 28 11:10:39]:c7ae9301-2d28-xxxx-xxxx-xxxxxxxxxxxx Operation-Uuid=19de28c6-0c89-4257-a4eb-22ae71dc71b1 Group=flexgroup Operation-Cookie=0 action=Start source=svm1:vol1__0007 destination=dr_svm1:dr_vol1__0007
Wed Sep 28 11:24:25 2022 Initialize[Sep 28 11:10:39]:c7ae92cc-2d28-xxxx-xxxx-xxxxxxxxxxxx Operation-Uuid=56292615-abbe-4845-8029-0fd3c5bc518f Group=flexgroup Operation-Cookie=0 action=End source=svm1:vol1__0005 destination=dr_svm1:dr_vol1__0005 status=Success bytes_transferred=78372119299 network_compression_ratio=1.0:1 transfer_desc=Logical Transfer with Storage Efficiency - Optimized Directory Mode
Wed Sep 28 11:24:36 2022 Initialize[Sep 28 11:10:39]:c7ae9301-2d28-xxxx-xxxx-xxxxxxxxxxxx Operation-Uuid=19de28c6-0c89-4257-a4eb-22ae71dc71b1 Group=flexgroup Operation-Cookie=0 action=End source=svm1:vol1__0007 destination=dr_svm1:dr_vol1__0007 status=Success bytes_transferred=76793484198 network_compression_ratio=1.0:1 transfer_desc=Logical Transfer with Storage Efficiency - Optimized Directory Mode
Wed Sep 28 11:24:50 2022 Initialize[Sep 28 11:10:39]:c7ae928e-2d28-xxxx-xxxx-xxxxxxxxxxxx Operation-Uuid=6b99cdd2-3e2a-490c-966c-826edfb01a06 Group=flexgroup Operation-Cookie=0 action=End source=svm1:vol1__0003 destination=dr_svm1:dr_vol1__0003 status=Success bytes_transferred=76364323106 network_compression_ratio=1.0:1 transfer_desc=Logical Transfer with Storage Efficiency - Optimized Directory Mode
Wed Sep 28 11:25:04 2022 Initialize[Sep 28 11:10:39]:c7ae923e-2d28-xxxx-xxxx-xxxxxxxxxxxx Operation-Uuid=75dcc6b4-1f2c-4830-a878-7daf2cf240d0 Group=flexgroup Operation-Cookie=0 action=End source=svm1:vol1__0001 destination=dr_svm1:dr_vol1__0001 status=Success bytes_transferred=89941005112 network_compression_ratio=1.0:1 transfer_desc=Logical Transfer with Storage Efficiency - Optimized Directory Mode
Wed Sep 28 11:25:04 2022 FlexGroupInitialize[Sep 28 11:10:36]:c7b4e440-2d28-xxxx-xxxx-xxxxxxxxxxxx Operation-Uuid=bd7135bc-3ed2-11ed-b811-d039ea995e93 Group=flexgroup Operation-Cookie=0 action=End source=svm1:vol1 destination=dr_svm1:dr_vol1 status=Failure message=Group Update failed(Failed to complete update operation on one or more item relationships.)
SnapMirror-error-log:
Wed Sep 28 11:25:04 2022 The following error code repeated 140 times cid=101, mid=923, Msg=Volume dr_svm1:dr_vol1 is not initialized.
Wed Sep 28 11:25:04 2022 FlexGroupInitialize[Sep 28 11:10:36]:c7b4e440-2d28-xxxx-xxxx-xxxxxxxxxxxx Operation-Uuid=bd7135bc-3ed2-11ed-b811-d039ea995e93 Group=flexgroup Operation-Cookie=0 SrcPath:svm1:vol1 DstPath:dr_svm1:dr_vol1 Prim=SM:cid=101,mid=2309, sm_wf_cg_group_op_wait_for_all_item_done_xfer_post_dowork:5741, Msg=Group Update failed Sec=SM:cid=101,mid=2308, sm_wf_common_post_action_gwf_working_set_errors:24890, Msg=Failed to complete update operation on one or more item relationships.
Wed Sep 28 11:25:04 2022 -:c7b4e440-2d28-xxxx-xxxx-xxxxxxxxxxxx Operation-Uuid=bd7135bc-3ed2-11ed-b811-d039ea995e93 Group=flexgroup Operation-Cookie=0 Trace_Buf (1 of 1 log entries): (EVT#762);(EVT#770);(EVT#771);(EVT#5);(EVT#808);(EVT#6);(EVT#815);(EVT#817);(EVT#818);(EVT#819);(EVT#820);(EVT#821);(EVT#822);(EVT#823);(EVT#774);(EVT#778);(EVT#5);(EVT#779);(EVT#780);(EVT#6);(EVT#781);(EVT#783);VUFCSd:common_s_:-1:_d_:-1_f_:0;(EVT#785);[CVId::Src-NE];[CVId::target-weekly.2022-09-18_0015];(EVT#805);(EVT#814);(EVT#873);(EVT#874);(EVT#875);(EVT#876);(EVT#877);(EVT#878);(EVT#887);(p6621445s6621444);(EVT#285);(p6621445s6621444);(EVT#288);
Wed Sep 28 11:31:32 2022 Release destination=dr_svm1:dr_vol1 release_mirror_imp:1564 Message=Relationship with destination dr_svm1:dr_vol1 not found.
EMS
Wed Sep 28 10:49:16 +0900 [cluster2-01: mgwd: cpeer.unavailable:alert]: Peer cluster cluster1 is no longer available.
Wed Sep 28 10:59:28 +0900 [cluster2-01: mgwd: cpeer.unavailable:alert]: Peer cluster cluster1 is no longer available.
Wed Sep 28 11:12:21 +0900 [cluster2-01: mgwd: cpeer.unavailable:alert]: Peer cluster cluster1 is no longer available.
Wed Sep 28 11:24:43 +0900 [cluster2-01: mgwd: cpeer.unavailable:alert]: Peer cluster cluster1 is no longer available.
Wed Sep 28 11:34:50 +0900 [cluster2-01: mgwd: cpeer.unavailable:alert]: Peer cluster cluster1 is no longer available.
Wed Sep 28 12:00:10 +0900 [cluster2-01: mgwd: cpeer.unavailable:alert]: Peer cluster cluster1 is no longer available.
EMS(フル)
Wed Sep 28 12:00:10 [cluster2-01: mgwd: cpeer.addr.stable.down:error]: Stable address xxx.xxx.x.146 of cluster cluster1, in IPspace Default, cannot be contacted.
Wed Sep 28 12:00:10 [cluster2-01: mgwd: cpeer.unavailable:alert]: Peer cluster cluster1 is no longer available.
Wed Sep 28 12:00:24 [cluster2-01: wafl_exempt04: wafl.vol.snap_create.done:info]: params: {'vol': 'vol0', 'app': '', 'volident': '', 'run_time': '135', 'owner': '', 'type': 'Volume'}
Wed Sep 28 12:00:24 [cluster2-01: wafl_exempt10: wafl.vol.blks_used.done:info]: params: {'vol': 'vol0', 'app': '', 'volident': '', 'run_time': '236', 'owner': '', 'type': 'Volume'}
Wed Sep 28 12:00:24 [cluster2-01: scan_ownblocks_calc_wkr: wafl.scan.ownblocks.done:info]: Completed block ownership calculation on volume vol0. The scanner took 101 ms.
Wed Sep 28 12:02:42 [cluster2-01: mgwd: cpeer.xcm.addr.disc.warn:error]: Address discovery failed for peer cluster xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. Reason: Failed to discover remote addresses: Failed to contact peer cluster "cluster1" at addresses: xxx.xxx.x.146, xxx.xxx.x.147. RPC: Remote system error [from mgwd on node "cluster2-01" (VSID: -3) to mgwd at xxx.xxx.x.146, xxx.xxx.x.147].
Wed Sep 28 12:02:42 [cluster2-01: mgwd: cpeer.xcm.update.warn:error]: Periodic update of peer network information failed. The following operations are incomplete: discovery failure.
Wed Sep 28 12:02:42 [cluster2-01: mgwd: mgmtgwd.jobmgr.private.jobcomplete.failure:info]: Private job "Cross-Cluster Manager" [id 109] (Cross-Cluster Address Manager) completed unsuccessfully: discovery failure (1).
Wed Sep 28 12:04:25 [cluster2-01: vifmgr: vifmgr.lifmoved.linkdown:notice]: LIF cluster2-01_mgmt1 (on virtual server 4294967295), IP address xxx.xxx.x.156, is being moved to node cluster2-01, port e0M.
Wed Sep 28 12:04:25 [cluster2-01: vifmgr: vifmgr.lifmoved.linkdown:notice]: LIF c1_lif_intercluster (on virtual server 4294967295), IP address xxx.xxx.x.148, is being moved to node cluster2-01, port e0e.
Wed Sep 28 12:05:00 [cluster2-01: wafl_exempt00: wafl.vol.snap_create.done:info]: params: {'vol': 'dr_svm1_root', 'app': '', 'volident': '@vserver:ab3b8af7-2d26-11ed-9a8a-d039ea995e93', 'run_time': '41', 'owner': '', 'type': 'Volume'}
Wed Sep 28 12:05:00 [cluster2-01: wafl_exempt03: wafl.vol.blks_used.done:info]: params: {'vol': 'dr_svm1_root', 'app': '', 'volident': '@vserver:ab3b8af7-2d26-11ed-9a8a-d039ea995e93', 'run_time': '234', 'owner': '', 'type': 'Volume'}
Wed Sep 28 12:05:00 [cluster2-01: scan_ownblocks_calc_wkr: wafl.scan.ownblocks.done:info]: Completed block ownership calculation on volume dr_svm1_root@vserver:ab3b8af7-2d26-11ed-9a8a-d039ea995e93. The scanner took 3 ms.
Wed Sep 28 12:05:00 [cluster2-01: sm_logger_main: smc.snapmir.update.fail:error]: Snapmirror update from source volume 'svm1:vol1' to destination volume 'dr_svm1:dr_vol1' failed with error 'Volume dr_svm1:dr_vol1 is not initialized.'. Relationship UUID 'c7b4e440-2d28-11ed-9a8a-d039ea995e93'.
Wed Sep 28 12:07:43 [cluster2-01: mgwd: cpeer.xcm.addr.disc.warn:error]: Address discovery failed for peer cluster xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. Reason: Failed to discover remote addresses: Failed to contact peer cluster "cluster1" at addresses: xxx.xxx.x.146, xxx.xxx.x.147. RPC: Remote system error [from mgwd on node "cluster2-01" (VSID: -3) to mgwd at xxx.xxx.x.146, xxx.xxx.x.147].
Wed Sep 28 12:07:43 [cluster2-01: mgwd: cpeer.xcm.update.warn:error]: Periodic update of peer network information failed. The following operations are incomplete: discovery failure.
Wed Sep 28 12:07:43 [cluster2-01: mgwd: mgmtgwd.jobmgr.private.jobcomplete.failure:info]: Private job "Cross-Cluster Manager" [id 109] (Cross-Cluster Address Manager) completed unsuccessfully: discovery failure (1).
Wed Sep 28 12:09:34 [cluster2-01: vifmgr: vifmgr.lifmoved.linkdown:notice]: LIF cluster2-01_mgmt1 (on virtual server 4294967295), IP address xxx.xxx.x.156, is being moved to node cluster2-01, port e0M.
Wed Sep 28 12:09:34 [cluster2-01: vifmgr: vifmgr.lifmoved.linkdown:notice]: LIF c1_lif_intercluster (on virtual server 4294967295), IP address xxx.xxx.x.148, is being moved to node cluster2-01, port e0e.
Wed Sep 28 12:12:44 [cluster2-01: mgwd: cpeer.xcm.addr.disc.warn:error]: Address discovery failed for peer cluster xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. Reason: Failed to discover remote addresses: Failed to contact peer cluster "cluster1" at addresses: xxx.xxx.x.146, xxx.xxx.x.147. RPC: Remote system error [from mgwd on node "cluster2-01" (VSID: -3) to mgwd at xxx.xxx.x.146, xxx.xxx.x.147].
Wed Sep 28 12:12:44 [cluster2-01: mgwd: cpeer.xcm.update.warn:error]: Periodic update of peer network information failed. The following operations are incomplete: discovery failure.
Wed Sep 28 12:12:44 [cluster2-01: mgwd: mgmtgwd.jobmgr.private.jobcomplete.failure:info]: Private job "Cross-Cluster Manager" [id 109] (Cross-Cluster Address Manager) completed unsuccessfully: discovery failure (1).
FlexGroup ボリュームリスト:
cluster1-01
Filesystem kbytes used avail capacity Mounted on
/vol/vol1__0003/ 7650410496 74188944 7576221552 1% /vol/vol1__0003/
/vol/vol1__0003/.snapshot 402653184 49320 402603864 0% /vol/vol1__0003/.snapshot
/vol/vol1__0005/ 7650410496 76492416 7573918080 1% /vol/vol1__0005/
/vol/vol1__0005/.snapshot 402653184 46624 402606560 0% /vol/vol1__0005/.snapshot
/vol/vol1__0001/ 7650410496 87232012 7563178484 1% /vol/vol1__0001/
/vol/vol1__0001/.snapshot 402653184 49064 402604120 0% /vol/vol1__0001/.snapshot
/vol/vol1__0007/ 7650410496 75001580 7575408916 1% /vol/vol1__0007/
/vol/vol1__0007/.snapshot 402653184 48440 402604744 0% /vol/vol1__0007/.snapshot
cluster1-02
Filesystem kbytes used avail capacity Mounted on
/vol/vol1__0008/ 7650410496 66892732 7583517764 1% /vol/vol1__0008/
/vol/vol1__0008/.snapshot 402653184 35872 402617312 0% /vol/vol1__0008/.snapshot
/vol/vol1__0002/ 7650410496 66263236 7584147260 1% /vol/vol1__0002/
/vol/vol1__0002/.snapshot 402653184 35588 402617596 0% /vol/vol1__0002/.snapshot
/vol/vol1__0004/ 7650410496 80140388 7570270108 1% /vol/vol1__0004/
/vol/vol1__0004/.snapshot 402653184 34776 402618408 0% /vol/vol1__0004/.snapshot
/vol/vol1__0006/ 7650410496 71515444 7578895052 1% /vol/vol1__0006/
/vol/vol1__0006/.snapshot 402653184 35908 402617276 0% /vol/vol1__0006/.snapshot
Cluster2-01
Filesystem kbytes used avail capacity Mounted on
/vol/dr_vol1__0003/ --- --- --- ---% /vol/dr_vol1__0003/
/vol/dr_vol1__0007/ --- --- --- ---% /vol/dr_vol1__0007/
/vol/dr_vol1__0005/ --- --- --- ---% /vol/dr_vol1__0005/
/vol/dr_vol1__0001/ --- --- --- ---% /vol/dr_vol1__0001/
cluster2 - 02
Filesystem kbytes used avail capacity Mounted on
/vol/dr_vol1__0004/ --- --- --- ---% /vol/dr_vol1__0004/
/vol/dr_vol1__0008/ --- --- --- ---% /vol/dr_vol1__0008/
/vol/dr_vol1__0002/ --- --- --- ---% /vol/dr_vol1__0002/
/vol/dr_vol1__0006/ --- --- --- ---% /vol/dr_vol1__0006/