MissingTokenError when "Attempting to retrieve auth token"
Applies to
- NetApp Element software 11.3 & 11.5 & 12.0 & 12.2
- Management node (mNode)
Issue
- Below outputs observed at the AIQ Collector container (mnode-svc-aiq-collector) logs:
<DATE TIME>:MainThread:[sf.mon.mnode_api_client:get_mnode_settings:101]INFO:Attempting to get session from mnode API
<DATE TIME>:MainThread:[sf.mon.mnode_api_client:log_it:36]DEBUG:Retrying sf.mon.mnode_api_client.MNodeApiClient.get_mnode_settings_with_retries in 60.0 seconds as it raised (Unable to decrypt stored MNode auth configuration settings. The data protection keys may have been removed or overwritten.)
<DATE TIME> INFO Attempting to retrieve auth token...
/tmp/_MEIpLgicD/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/la...l#ssl-warnings
<DATE TIME> ERROR (missing_token) Missing access token parameter.
Traceback (most recent call last):
File "mnode-aiq-collector.py", line 390, in main
File "mnode-aiq-collector.py", line 30, in startup
File "mnode-aiq-collector.py", line 359, in _get_mnode_service_session
File "mnode-aiq-collector.py", line 339, in _get_client_credentials_session
File "site-packages/requests_oauthlib/oauth2_session.py", line 244, in fetch_token
File "site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 411, in parse_request_body_response
File "site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 379, in parse_token_response
File "site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 389, in validate_token_parameters
MissingTokenError: (missing_token) Missing access token parameter.
- xDBVersionMismatch errors when calling SetAuthConfiguration will be shown at the new cluster master's auth log (
/var/log/sf-element-auth.info
)
<DATE TIME> NODENAME element-auth[22660]: An error occurred while refreshing the key ring. Will try again in 2 minutes.
<DATE TIME> NODENAME element-auth[22660]: element_auth_svc.Exceptions.ServiceUnavailableException: Unable to store configuration data to Element API: ms/MSRestAPIServer.cpp:951
6 void solidfire::MSRestAPIServer::SetAuthConfiguration(const solidfire::JsonParams&, Json::Value&): Check failed: [Throw]
<DATE TIME> NODENAME element-auth[22660]: Exception: xDBVersionMismatch at void solidfire::ThrowIfFatalError(const solidfire::DBException&, const string&) (cs/DBCommon.cpp:134): DBClient operation failed due to version mismatch at [/elementauth/mNode]
<DATE TIME> NODENAME element-auth[22660]: ---> Element.Core.Exceptions.ApiServerException: ms/MSRestAPIServer.cpp:9516 void solidfire::MSRestAPIServer::SetAuthConfiguration(const solidfire::JsonParams&, Json::Value&): Check failed: [Throw]
<DATE TIME> NODENAME element-auth[22660]: Exception: xDBVersionMismatch at void solidfire::ThrowIfFatalError(const solidfire::DBException&, const string&) (cs/DBCommon.cpp:134): DBClient operation failed due to version mismatch at [/elementauth/mNode]
- Swagger UI cannot login or authenticate with the cluster
- Swagger UI gives
Internal Server error
- Hybrid Cloud Control (HCC) UI gives 500 connection error
- HCC UI cannot authenticate with the cluster
- mnode-svc-aiq-collector service shows following errors repeatedly:
mnode_mnode-svc-aiq-collector.1.ldqo22pevihd@SF-F9D9 | Sep 12 2020 14:58:49.620 INFO Attempting to retrieve asset information...
mnode_mnode-svc-aiq-collector.1.wli9iaea4d1d@SF-F9D9 | Sep 12 2020 10:31:51.407 INFO Attempting to retrieve asset information...