StorageGRID S3のレイテンシとCassandraメトリックの増加

環境

• StorageGRID アプライアンス
• ベアメタル・ノード

問題

• S3 レイテンシの増加

• Cassandraリモートメッセージレイテンシの増加

• Cassandra でドロップされたメッセージの増加

• Cassandraタイムアウトの増加

• Cassandra保留中のコンパクションタスクの増加

• Compactor Executorの例外：

// Exception

ERROR [CompactionExecutor:12245] 2023-05-23 20:01:51,515 CassandraDaemon.java (line 231) Exception in thread Thread[CompactionExecutor:12245,5,main]
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
     at org.apache.cassandra.db.rows.BTreeRow$Builder$CellResolver.resolve(BTreeRow.java:649)
     at org.apache.cassandra.db.rows.BTreeRow$Builder$CellResolver.resolve(BTreeRow.java:610)
     at org.apache.cassandra.utils.btree.BTree$Builder.resolve(BTree.java:1082)
     at org.apache.cassandra.db.rows.BTreeRow$Builder.build(BTreeRow.java:790)
     at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:627)
     at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeOne(UnfilteredSerializer.java:480)
     at org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:436)
     at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:96)
     at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:74)
     at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
     at org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:159)
     at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
     at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
     at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
     at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133)
     at org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:113)
     at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173)
     at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:142)
     at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
     at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:145)
     at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:211)
     at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
     at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:88)
     at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
     at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:307)
     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
     at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
     at java.lang.Thread.run(Thread.java:748)