AIQUMサーバがパフォーマンスデータの表示を停止する
環境
ActiveIQ Unified Manager(UM)9.x
問題
- パフォーマンスグラフが完成していません
- メモリ使用率が高い
- 次のエラーが原因でジョブがスキップされます。また、強制取得もタイムアウトエラーで失敗します。
2024-01-11 06:24:29,863 INFO [oncommand] [org.springframework.jms.listener.DefaultMessageListenerContainer#2-1] [com.netapp.ipc.jms.OCIE_Events] OCIE JMS notification message received: {DatasourceName=172.2.71.70, DatasourceID=12, PackageName=netappstorageperformance, PollInterval=300000, NotificationType=PACKAGE_STARTED, PollStartTime=1704934443518, MessageType=PACKAGE_NOTIFICATION} 2024-01-11 06:29:03,610 INFO [oncommand] [org.springframework.jms.listener.DefaultMessageListenerContainer#2-1] [com.netapp.ipc.jms.OCIE_Events] OCIE JMS notification message received: {DatasourceName=172.2.71.70, DatasourceID=12, PackageName=netappstorageperformance, PollInterval=-1, NotificationType=PACKAGE_SKIPPED, PollStartTime=1704934743594, MessageType=PACKAGE_NOTIFICATION} 2024-01-11 06:29:03,610 INFO [oncommand] [org.springframework.jms.listener.DefaultMessageListenerContainer#2-1] [com.netapp.ipc.jms.OCIE_Events] Ignoring notification : PACKAGE_SKIPPED for datasource 172.2.71.70 : another POLL already running 2024-01-11 06:45:05,881 INFO [oncommand] [org.springframework.jms.listener.DefaultMessageListenerContainer#1-1] [com.netapp.ipc.jms.OCIE_Events] OCIE JMS notification message received: {DatasourceName=172.2.71.70, DatasourceID=12, PackageName=netappfoundation, NotificationType=PACKAGE_SKIPPED, PollStartTime=1704935705881, MessageType=PACKAGE_NOTIFICATION} 2024-01-11 06:45:05,881 INFO [oncommand] [reconciliation-0] [c.n.dfm.collector.OcieJmsListener] Ignoring notification : PACKAGE_SKIPPED for cluster 172.2.71.70 : another POLL already running 2024-01-11 06:45:05,881 INFO [oncommand] [org.springframework.jms.listener.DefaultMessageListenerContainer#1-1] [com.netapp.ipc.jms.OCIE_Events] Ignoring notification : PACKAGE_SKIPPED for datasource 172.2.71.70 : another POLL already running 2024-01-11 12:30:05,876 WARN [Timer-0] c.o.s.a.f.m.s.SchedulerTask (SchedulerTask.java:187) - 172.2.71.70 - Data source is already performing a poll of netappfoundation. Therefore this poll is skipped. 2024-01-11 12:34:03,528 WARN [Timer-0] c.o.s.a.f.m.s.SchedulerTask (SchedulerTask.java:187) - 172.2.71.70 - Data source is already performing a poll of netappstorageperformance. Therefore this poll is skipped. 2024-01-11 12:39:03,521 WARN [Timer-0] c.o.s.a.f.m.s.SchedulerTask (SchedulerTask.java:187) - 172.2.71.70 - Data source is already performing a poll of netappstorageperformance. Therefore this poll is skipped. 2024-01-11 17:46:25,230 INFO [clouduser] [job-468] [refresh(cluster=DR-NETAPP-CLUS)|START] [com.netapp.dfm.job.StartTask] Started job 2024-01-11 17:46:25,777 INFO [clouduser] [job-469] [refresh(cluster=DR-NETAPP-CLUS)|forceAcquisition] [c.n.d.c.OcieAcquisitionFacade] Forced acquisition completed for DataSource 172.2.71.70 (ID: 12) 2024-01-11 17:46:25,777 INFO [clouduser] [job-469] [refresh(cluster=DR-NETAPP-CLUS)|forceAcquisition] [c.n.d.c.ForceAcquisitionTask] Forced acquisition at 17:46:25.293. Waiting for next changelog update. 2024-01-11 19:46:25,643 ERROR [clouduser] [job-468] [refresh(cluster=DR-NETAPP-CLUS)|forceAcquisition] [com.netapp.dfm.job.JobEngine] Error executing job 1678d184741d3fe7:-58b60265:18cde886c56:-7f63 task 1678d184741d3fe7:-58b60265:18cde886c56:-7f65: Timed out after 02:00:00 while waiting on acquisition forced at 17:46:25.293 java.lang.RuntimeException: Timed out after 02:00:00 while waiting on acquisition forced at 17:46:25.293 at deployment.dfm-app.war//com.netapp.dfm.collector.InventoryChangeWaitTask.suspendIfNecessary(InventoryChangeWaitTask.java:201) at deployment.dfm-app.war//com.netapp.dfm.collector.InventoryChangeWaitTask.resume(InventoryChangeWaitTask.java:148) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at deployment.dfm-app.war//com.netapp.dfm.job.TaskExecuteInvocation.invoke(TaskExecuteInvocation.java:37) at deployment.dfm-app.war//com.netapp.dfm.job.JobEngine.executeTask(JobEngine.java:543) at deployment.dfm-app.war//com.netapp.dfm.job.JobEngine.process(JobEngine.java:513) at deployment.dfm-app.war//com.netapp.dfm.job.JobEngine.process(JobEngine.java:56) at deployment.dfm-app.war//com.netapp.dfm.job.JobEngine$$FastClassBySpringCGLIB$$4bdbd68e.invoke(<generated>) at deployment.dfm-app.war//org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at deployment.dfm-app.war//org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at deployment.dfm-app.war//org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at deployment.dfm-app.war//org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:704) at deployment.dfm-app.war//com.netapp.dfm.job.JobEngine$$EnhancerBySpringCGLIB$$56e5a541.process(<generated>) at deployment.dfm-app.war//com.netapp.dfm.workqueue.WorkerThread.run(WorkerThread.java:44) 2024-01-11 19:46:25,758 INFO [clouduser] [job-467] [refresh(cluster=DR-NETAPP-CLUS)|FINISH] [com.netapp.dfm.job.FinishTask] Completed job in 2:00:00.528 2024-01-11 19:46:34,881 ERROR [clouduser] [job-469] [clusterAcquisition(cluster=DR-NETAPP-CLUS)|refresh.wait.storageObject(DR-NETAPP-CLUS)] [com.netapp.dfm.job.JobEngine] Error executing job 1678d184741d3fe7:-58b60265:18cde886c56:-7f6e task 1678d184741d3fe7:-58b60265:18cde886c56:-7f6f: Monitoring refresh job ID 1678d184741d3fe7:-58b60265:18cde886c56:-7f63 for object 'DR-NETAPP-CLUS' completed unsuccessfully (Status: FAILED). java.lang.RuntimeException: Monitoring refresh job ID 1678d184741d3fe7:-58b60265:18cde886c56:-7f63 for object 'DR-NETAPP-CLUS' completed unsuccessfully (Status: FAILED). at deployment.dfm-app.war//com.netapp.dfm.protect.task.WaitForStorageObjectRefreshTask.execute(WaitForStorageObjectRefreshTask.java:78) at jdk.internal.reflect.GeneratedMethodAccessor919.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at deployment.dfm-app.war//com.netapp.dfm.job.TaskExecuteInvocation.invoke(TaskExecuteInvocation.java:37) at deployment.dfm-app.war//com.netapp.dfm.job.JobEngine.executeTask(JobEngine.java:543) at deployment.dfm-app.war//com.netapp.dfm.job.JobEngine.process(JobEngine.java:513) at deployment.dfm-app.war//com.netapp.dfm.job.JobEngine.process(JobEngine.java:56) at deployment.dfm-app.war//com.netapp.dfm.job.JobEngine$$FastClassBySpringCGLIB$$4bdbd68e.invoke(<generated>) at deployment.dfm-app.war//org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at deployment.dfm-app.war//org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at deployment.dfm-app.war//org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at deployment.dfm-app.war//org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:704) at deployment.dfm-app.war//com.netapp.dfm.job.JobEngine$$EnhancerBySpringCGLIB$$56e5a541.process(<generated>) at deployment.dfm-app.war//com.netapp.dfm.workqueue.WorkerThread.run(WorkerThread.java:44) 2024-01-11 19:46:34,922 INFO [clouduser] [job-470] [clusterAcquisition(cluster=DR-NETAPP-CLUS)|FINISH] [com.netapp.dfm.job.FinishTask] Completed job in 2:00:09.879