StorageGRIDバケットでの取り込み/読み出し時にエラー504ゲートウェイが断続的にタイムアウトする

最後の更新
PDFとして保存

Views:: 1

Visibility:: Public

Votes:: 0

Category:: storagegrid<a>2009994925</a>

Specialty:: sgrid

Last Updated:

環境

StorageGRID OS 11.7.0

問題

StorageGRIDバケットからのオブジェクトの取り込み/読み出しで一時的に発生するエラー：

Error: reading blob <>: fetching blob: received unexpected HTTP status: 504 Gateway Time-out

StorageGRID Graphanaでは 504499 、アプリケーション側から観測された問題と一致するゲートウェイノードのエラーと一致するエラーを確認しました。

Failed to pull docker image from bucket: Error timed out

デバッグレベルのロギングを有効にする
StorageGRIDのログから、クライアントからストレージノードへゲートウェイノード経由での要求のアップストリーム時に、SSLハンドシェイクタイムアウトが発生した問題を特定しました。

ゲートウェイノード- /var/local/log/nginx-gw/access.log

2024/04/15 06:40:08 [warn] 1327055#1327055: *580890363 [lua] sgbalancer.lua:564: on_balance_phase(): Connection to 47f9ba08-bc36-4be5-bfe3-cde14b40f83f failed with code 504 in worker 23 while connecting to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-01-registry?list-type=2&max-keys=1&prefix=, host: "<Gateway node>:10443"

2024/04/15 06:40:13 [warn] 1327099#1327099: *580934305 [lua] sgbalancer.lua:564: on_balance_phase(): Connection to 47f9ba08-bc36-4be5-bfe3-cde14b40f83f failed with code 504 in worker 31 while connecting to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?max-keys=1&prefix=, host: "<Gateway node>:10443"

2024/04/15 08:19:56 [warn] 1327060#1327060: *581032520 [lua] sgbalancer.lua:564: on_balance_phase(): Connection to 47f9ba08-bc36-4be5-bfe3-cde14b40f83f failed with code 504 in worker 24 while connecting to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix=docker%2Fregistry%2Fv2%2Fblobs%2Fsha256%2F38%2F38c364755503d3fa4adc039af63e5108189b759cf6ad9d26f7219072c427db83%2Fdata HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?list-type=2&max-keys=1&prefix=docker%2Fregistry%2Fv2%2Fblobs%2Fsha256%2F38%2F38c364755503d3fa4adc039af63e5108189b759cf6ad9d26f7219072c427db83%2Fdata, host: "<Gateway node>:10443"

2024/04/15 06:40:08 [error] 1327055#1327055: *580890363 upstream timed out (110: Connection timed out) while SSL handshaking to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?list-type=2&max-keys=1&prefix=, host: "<Gateway node>:10443"

2024/04/15 08:34:50 [error] 1327060#1327060: *581032520 upstream timed out (110: Connection timed out) while SSL handshaking to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?list-type=2&max-keys=1&prefix=, host: "<Gateway node>:10443"