StorageGRID バケットでの取り込み / 読み出し中の断続的なエラー 504 Gateway Time-out

最後の更新
PDFとして保存

Views:: 6

Visibility:: Public

Votes:: 0

Category:: storagegrid

Specialty:: sgrid

Last Updated:

環境

StorageGRID OS 11.7.0

問題

StorageGRID バケットからのオブジェクトの取り込み / 読み出しで断続的なエラーが発生する：

Error: reading blob <>: fetching blob: received unexpected HTTP status: 504 Gateway Time-out

StorageGRID Graphanaから、ゲートウェイノードの499エラーと一致する504エラーが観測され、これはアプリケーション側から観測された問題と一致します：

Failed to pull docker image from bucket: Error timed out

デバッグレベルのロギングを有効化
StorageGRIDのログから、クライアントからゲートウェイノード経由でストレージノードへの要求のアップストリーム中にSSLハンドシェイクがタイムアウトするという問題を特定しました：

ゲートウェイノード - /var/local/log/nginx-gw/access.log

2024/04/15 06:40:08 [warn] 1327055#1327055: *580890363 [lua] sgbalancer.lua:564: on_balance_phase(): Connection to 47f9ba08-bc36-4be5-bfe3-cde14b40f83f failed with code 504 in worker 23 while connecting to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-01-registry?list-type=2&max-keys=1&prefix=, host: "<Gateway node>:10443"

2024/04/15 06:40:13 [warn] 1327099#1327099: *580934305 [lua] sgbalancer.lua:564: on_balance_phase(): Connection to 47f9ba08-bc36-4be5-bfe3-cde14b40f83f failed with code 504 in worker 31 while connecting to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?max-keys=1&prefix=, host: "<Gateway node>:10443"

2024/04/15 08:19:56 [warn] 1327060#1327060: *581032520 [lua] sgbalancer.lua:564: on_balance_phase(): Connection to 47f9ba08-bc36-4be5-bfe3-cde14b40f83f failed with code 504 in worker 24 while connecting to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix=docker%2Fregistry%2Fv2%2Fblobs%2Fsha256%2F38%2F38c364755503d3fa4adc039af63e5108189b759cf6ad9d26f7219072c427db83%2Fdata HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?list-type=2&max-keys=1&prefix=docker%2Fregistry%2Fv2%2Fblobs%2Fsha256%2F38%2F38c364755503d3fa4adc039af63e5108189b759cf6ad9d26f7219072c427db83%2Fdata, host: "<Gateway node>:10443"

2024/04/15 06:40:08 [error] 1327055#1327055: *580890363 upstream timed out (110: Connection timed out) while SSL handshaking to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?list-type=2&max-keys=1&prefix=, host: "<Gateway node>:10443"

2024/04/15 08:34:50 [error] 1327060#1327060: *581032520 upstream timed out (110: Connection timed out) while SSL handshaking to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?list-type=2&max-keys=1&prefix=, host: "<Gateway node>:10443"