Hadoopアプリケーションクエリの実行とStorageGRIDバケットへの書き込みが失敗し、タイムアウトが発生する
環境
- StorageGRID 11.6.0.13
- Hadoop
問題
- Hadoop アプリケーションクエリ の実行が失敗し、クエリがバケットへの書き込みに失敗しているためタイムアウトします。
- 送信元アプリケーションエラー：
botocore.exceptions.Client
2024-01-24 17:11:54.335 DEBUG com.amazonaws.thirdparty.apache. http.wire: 'main': http-outgoing-1 <<"<Error><Code>ServiceUnavailable</Code><Message>Please reduce your request rate.</Message><Resource>/<BUCKET>/?uploads&prefix=daXXXrmediate%2Fquery_test2%2F</Resource><RequestId></RequestId></Error>"
2024-01-24 17:11:54.335 DEBUG com.amazonaws.thirdparty.apache. http.headers: 'main': http-outgoing-1 << HTTP/1.1 503 Service Unavailable- StorageGRID ログから：
Jan 25 08:46:07 XXXNODE04 ADE: |12330832 2577485089 S3RQ %CEA 2024-01-25T08:46:07.470023| NOTICE 1448 3018ddf97d08def3 S3RQ: EVENT_PROCESS_CREATE - connection=17061XXX127 method=GET name=</<BUCKET>/?uploads&prefix=datalake_intermediate%2Fquery_test%2F> auth=<V4> clientIP=<>
Jan 25 08:47:01 XXXNODE04 ADE: |12330832 2577485089 S3RQ CSEN 2024-01-25T08:47:01.869914| WARNING 0236 3018ddf97d08def3 S3RQ: S3Request_Module::processCreate Cassandra TimeoutException (posSuc: Failed to execute statement while paging. - Cassandra Driver Error(Request timed out):'Request timed out' Detailed Info:[N/A])
Jan 25 08:47:01 XXXNODE04 ADE: |12330832 2577485089 S3RQ CSEN 2024-01-25T08:47:01.869996| NOTICE 0089 3018ddf97d08def3 S3RQ: S3 error response: RequestId=1706XXX65127, TraceId=3018ddf97d08def3, Resource=/<BUCKET>/?uploads&prefix=daXXXmediate%2Fquery_test%2F, HTTP Method=GET, HTTP Status Code=503, X-Forwarded-For: 'XX.XX.XXX.91', ErrorMsg=ServiceUnavailable, ErrorType=Internal, CustomErrorMessage={<none>}, Details={posSuc: Failed to execute statement while paging. - Cassandra Driver Error(Request timed out):'Request timed out' Detailed Info:[N/A]}