動的なしきい値を持つONTAPバージョンでの修正可能なメモリエラーの報告

環境

ONTAPバージョン：

• 9.5以降のメジャーリリース
• 9.4P6以降のPリリース
• 9.3P11以降のPリリース
• 9.1P18以降のPリリース

プラットフォーム：

• AFFシステム
• ASAシステム
• FASシステム

• CriticalCECCCountMemErrAlert

注：その他のすべてのONTAPバージョンについては、How to troubleshoot correctable memory errors on FAS and AFF systems

問題

• ヘルスアラート AutoSupport メッセージ:

HA Group Notification (Health Monitor process nphm: CriticalCECCCountMemErrAlert[DIMM-16]) ALERT

• EMSは報告しています：

[mgwd: callhome.hm.alert.critical:alert]: Call home for Health Monitor process nphm: CriticalCECCCountMemErrAlert[DIMM-16].

• 管理ログ

[nphmd: hm.alert.raised:alert]: Alert Id = CriticalCECCCountMemErrAlert , Alerting Resource = DIMM-16 raised by monitor controller

• Health Monitorレポート

::> system health alert show
        Node: Node1
      Alert ID: CriticalCECCCountMemErrAlert
      Resource: DIMM-1
      Severity: Critical
   Indication Time: Wed Jan 10 16:45:18 2024
      Suppress: false
     Acknowledge: false
   Probable Cause: The DIMM has degraded, leading to memory errors.
  Possible Effect: Memory issues can lead to a catastrophic system panic,
           which can lead to data downtime on the node.
 Corrective Actions:
 1. Contact technical support to obtain a new DIMM of the same specification.
 2. If possible, perform a takeover of this node and bring the node down for maintenance.
 3. Refer to the DIMM replacement guide for your given hardware platform to replace the DIMM.
 4. Bring the storage system online.

Alertdditional Information  
 Slot Name: DIMM-16
 CPU Socket: 0
 Channel: 0
 Slot Number on a Channel: 0
 Correctable ECC Error Count: 329793
 Uncorrectable ECC Error Count: 0
 Correctable ECC Error Limit: 0
 CECC Alert Method: bucket
 Correctable ECC Replace DIMM: true
 Failure Reason: none
 Correctable ECC Error Minimum Count: 325104
 Correctable ECC Error Bucket Count: 177243
 Correctable ECC Replace DIMM Count: 3
 DIMM Serial Number: *****
 Node Serial Number: *****
 Node Model: FAS****
 Alerting Resource Name   DIMM-16

• Memory dimm showは、dimmのステータスをdegradedと報告します

Cluster::>memory dimm show
              DIMM     UECC  CECC  Alert    CPU           Slot          Failure
Node          Name    Count Count Method Socket Channel Number Status    Reason
------------- ------- ----- ----- ------ ------ ------- ------ ------- --------
Cluster-01
              DIMM-1      0     0 bucket      1       0      0 ok          none
              DIMM-2      0     0 bucket      1       0      1 ok          none
              DIMM-3      0     0 bucket      1       1      0 ok          none
              DIMM-4      0     0 bucket      1       1      1 ok          none
              DIMM-5      0     0 bucket      1       3      1 ok          none
              DIMM-6      0     0 bucket      1       3      0 ok          none
              DIMM-7      0     0 bucket      1       2      1 ok          none
              DIMM-8      0     0 bucket      1       2      0 ok          none
              DIMM-9      0     0 bucket      0       2      0 ok          none
              DIMM-10     0     0 bucket      0       2      1 ok          none
              DIMM-11     0     0 bucket      0       3      0 ok          none
              DIMM-12     0     0 bucket      0       3      1 ok          none
              DIMM-13     0     0 bucket      0       1      1 ok          none
              DIMM-14     0     0 bucket      0       1      0 ok          none
              DIMM-15     0     0 bucket      0       0      1 ok          none
              DIMM-16     0     0 bucket      0       0      0 degraded   none
 

• sktrace にエラーが表示されると、AutoSupport ログでパフォーマンスの問題が発生することがあります

CECC_LOG_ENTRY: Number of correctable ECC since boot 355314194: Information about Correctable ECC: ECC error at DIMM 8: CE-12-1234-12345678,ADDR 0xaada0eec0, (Node(0), Memory controller(1), CH(2), DIMM(0), Rank(0), Bank Group(3), Bank(0x2), Row(0x2174), Col(0x1d8))