Script:收集Enterprise Manager Grid Control/Agent/Database Control诊断信息

以下脚本可以提供一个Enterprise Manager Grid Control/Agent/Database Control发生故障时诊断问题,收集有用信息日志的思路; 显然大多数人对Grid Control/OMS 、 Agent 、 DB Console一些关键日志的存放路径都不太了解, 有这样一个大纲 就清晰多了:

 

-Grid Control/OMS

EMDIAG Kit : EMDIAG是一套收集Grid Control或DB Control诊断信息的工具
(Extracts data from the Grid Control or DB Control Repository to help you to maintain or diagnose problems
in your Grid Control or DB Control.);具体可以参考MOS文档EMDIAG Master Index [ID 421053.1]

   - emctl status oms
   - emctl getversion oms
   - emctl status blackout
   - emctl secure status oms
   - emctl dump omsthread
   - <OMS的ORACLE_HOME> 的补丁和版本信息
   - <OMS的ORACLE_HOME>/sysman/config/emoms.properties
   - <OMS的ORACLE_HOME>/sysman/config/emomslogging.properties
   - <OMS的ORACLE_HOME>/sysman/setupinfo.txt
   - <OMS的ORACLE_HOME>/sysman/admin/nsupportedtzs.lst
   - <OMS的ORACLE_HOME>/sysman/admin/supportedtzs.lst
   - <OMS的ORACLE_HOME>/sysman/admin/tzmappings.lst
   - <OMS的ORACLE_HOME>/sysman/admin/discover/discover.lst
   - <OMS的ORACLE_HOME>/sysman/config/httpd_em.conf
   - <OMS的ORACLE_HOME>/sysman/config/*.properties
   - <OMS的ORACLE_HOME>/sysman/emd/*.ini
   - <OMS的ORACLE_HOME>/sysman/emd/*.xml
   - <OMS的ORACLE_HOME>/sysman/recv 
   - <OMS的ORACLE_HOME>/j2ee/OC4J_EM/log/*
   - <OMS的ORACLE_HOME>/j2ee/OC4J_EM/log/OC4J_EM_default_island_1/*
   - <OMS的ORACLE_HOME>/cfgtoollogs/
   - <OMS的ORACLE_HOME>/cfgtoollogs/cfgfw/*.log
   - <OMS的ORACLE_HOME>/cfgtoollogs/oui/*.log
   - <OMS的ORACLE_HOME>/cfgtoollogs/oui/*.err
   - <OMS的ORACLE_HOME>/cfgtoollogs/oui/*.out
   - <OMS的ORACLE_HOME>/cfgtoollogs/opatch/*.txt
   - <OMS的ORACLE_HOME>/cfgtoollogs/opatch/*.log
   - <OMS的ORACLE_HOME>/Apache/Apache/conf/*.conf
   - <OMS的ORACLE_HOME>/config/*
   - <OMS的ORACLE_HOME>/chgip/config/*
   - <OMS的ORACLE_HOME>/chgip/log/*
   - <OMS的ORACLE_HOME>/chgip/scripts/*
   - <OMS的ORACLE_HOME>/dcm/config/*
   - <OMS的ORACLE_HOME>/dcm/logs/*
   - <OMS的ORACLE_HOME>/install/*
   - <OMS的ORACLE_HOME>/j2ee/OC4J_EM/application-deployments/em/em/orion-web.xml
   - <OMS的ORACLE_HOME>/j2ee/OC4J_EM/applications/em/em/WEB-INF/uix-config.xml
   - <OMS的ORACLE_HOME>/j2ee/OC4J_EM/applications/em/em/WEB-INF/web.xml
   - <OMS的ORACLE_HOME>/opmn/logs/*
   - <OMS的ORACLE_HOME>/opmn/conf/*.xml
   - <OMS的ORACLE_HOME>/opmn/conf/*.conf

Agent

   - emctl status agent
   - emctl upload
   - emctl status agent scheduler
   - emctl status agent memory
   - emctl status agent jobs
   - emctl config agent getTZ
   - emctl status agent secure
   - emctl secure status agent
   - ping OMS服务器ip的结果
   - <Agent的ORACLE_HOME> 的补丁和版本信息 opatch -lsinventory
   - <Agent的ORACLE_HOME>/sysman/config/emd.properties
   - <Agent的ORACLE_HOME>/sysman/config/emagentlogging.properties
   - <Agent的ORACLE_HOME>/sysman/setupinfo.txt
   - <Agent的ORACLE_HOME>/sysman/admin/nsupportedtzs.lst
   - <Agent的ORACLE_HOME>/sysman/admin/supportedtzs.lst
   - <Agent的ORACLE_HOME>/sysman/admin/tzmappings.lst
   - <Agent的ORACLE_HOME>/sysman/admin/discover/discover.lst
   - <Agent的ORACLE_HOME>/sysman/config/*.properties
   - <Agent的ORACLE_HOME>/sysman/emd/*.ini
   - <Agent的ORACLE_HOME>/sysman/emd/*.xml
   - <Agent的ORACLE_HOME>/sysman/emd/
   - <Agent的ORACLE_HOME>/sysman/log/*
   - <Agent的ORACLE_HOME>/sysman/emd/*
   - <Agent的ORACLE_HOME>/cfgtoollogs/
   - <Agent的ORACLE_HOME>/cfgtoollogs/cfgfw/*.log
   - <Agent的ORACLE_HOME>/cfgtoollogs/oui/*.log
   - <Agent的ORACLE_HOME>/cfgtoollogs/oui/*.err
   - <Agent的ORACLE_HOME>/cfgtoollogs/oui/*.out
   - <Agent的ORACLE_HOME>/cfgtoollogs/opatch/*.txt
   - <Agent的ORACLE_HOME>/cfgtoollogs/opatch/*.log


Enterprise Manager Database Control

同样可以用EMDIAG Kit 收集diag info 

   - emctl status dbconsole
   - emctl status agent
   - emctl upload
   - emctl status agent scheduler
   - emctl status agent jobs
   - emctl config agent getTZ
   - emctl status agent
   - emctl status blackout
   - $ORACLE_HOME/sysman/emd/targets.xml
   - $ORACLE_HOME/<hostname>_<sid>/sysman/config/emoms.properties
   - $ORACLE_HOME/<hostname>_<sid>/sysman/config/emomslogging.properties
   - $ORACLE_HOME/<hostname>_<sid>/sysman/config/emd.properties
   - $ORACLE_HOME/<hostname>_<sid>/sysman/config/emagentlogging.properties
   - $ORACLE_HOME/sysman/admin/nsupportedtzs.lst
   - $ORACLE_HOME/sysman/admin/supportedtzs.lst
   - $ORACLE_HOME/sysman/admin/tzmappings.lst
   - $ORACLE_HOME/sysman/admin/discover/discover.lst
   - $ORACLE_HOME/<hostname>_<sid>/sysman/setupinfo.txt
   - $ORACLE_HOME/<hostname>_<sid>/config/*
   - $ORACLE_HOME/<hostname>_<sid>/emd/*.ini
   - $ORACLE_HOME/<hostname>_<sid>/emd/*.xml
   - $ORACLE_HOME/<hostname>_<sid>/emd/*.txt
   - $ORACLE_HOME/<hostname>_<sid>/emd/state/
   - $ORACLE_HOME/<hostname>_<sid>/emd/upload/
   - $ORACLE_HOME/<hostname>_<sid>/emd/recv/
   - $ORACLE_HOME/<hostname>_<sid>/sysman/log/*
   - $ORACLE_HOME/opmn/logs/*
   - $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_<hostname>_<sid>/log/*
   - $ORACLE_HOME/cfgtoollogs/*
   - $ORACLE_HOME/cfgtoollogs/emca/*
   - $ORACLE_HOME/cfgtoollogs/cfgfw/*
   - $ORACLE_HOME/cfgtoollogs/oui/*
   - $ORACLE_HOME/cfgtoollogs/opatch/*
   - $ORACLE_HOME/Apache/Apache/conf/*.conf
   - $ORACLE_HOME/install/*
   - $ORACLE_HOME/inventory/ContentsXML/comps.xml
   - $ORACLE_HOME/oc4j/j2ee/oc4j_applications/applications/em/em/WEB-INF/uix-config.xml
   - $ORACLE_HOME/oc4j/j2ee/oc4j_applications/applications/em/em/WEB-INF/web.xml
   - $ORACLE_HOME/opmn/conf/opmn.xml
   - $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_<hostname>_<sid>/application-deployments/default/defaultWebApp/*
   - $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_<hostname>_<sid>/application-deployments/em/*
   - $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_<hostname>_<sid>/application-deployments/em/em/*
   - $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_<hostname>_<sid>/config/*

Comments

  1. maclean says

    [emuser@nas ~]$ export OMS_HOME=/w01/wls/oms
    [emuser@nas ~]$ export AGENT_HOME=/w01/wls/agent/core/12.1.0.1.0
    [emuser@nas ~]$ $OMS_HOME/bin/emctl status oms
    Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
    Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.
    WebTier is Up
    Oracle Management Server is Up

    [emuser@nas ~]$ $OMS_HOME/bin/emctl getversion oms
    Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
    Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.
    Enterprise Manager 12c OMS Version 12.1.0.1.0

    [emuser@nas ~]$ $OMS_HOME/bin/emctl dump omsthread

    Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
    Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.
    Dumping thread for process ‘24001’
    2012-01-21 10:27:35
    Full thread dump Java HotSpot(TM) 64-Bit Server VM (19.1-b02 mixed mode):

    “Attach Listener” daemon prio=10 tid=0x000000005ea73000 nid=0x76c3 waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

    Locked ownable synchronizers:
    – None

    “Thread-229” daemon prio=10 tid=0x0000000060f57000 nid=0x71e6 runnable [0x00000000491af000]
    java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
    at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
    – locked (a java.lang.Object)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
    at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
    – locked (a com.sun.net.ssl.internal.ssl.AppInputStream)
    at oracle.ods.virtualization.engine.backend.jndi.OvdJndiInputStream.read(OvdJndiInputStream.java:59)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    – locked (a java.io.BufferedInputStream)
    at com.sun.jndi.ldap.Connection.run(Connection.java:808)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “Thread-228” daemon prio=10 tid=0x00000000607b0000 nid=0x71e5 runnable [0x00000000490ae000]
    java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
    at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
    – locked (a java.lang.Object)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
    at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
    – locked (a com.sun.net.ssl.internal.ssl.AppInputStream)
    at oracle.ods.virtualization.engine.backend.jndi.OvdJndiInputStream.read(OvdJndiInputStream.java:59)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    – locked (a java.io.BufferedInputStream)
    at com.sun.jndi.ldap.Connection.run(Connection.java:808)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “Thread-227” daemon prio=10 tid=0x00000000615ff800 nid=0x71e4 runnable [0x0000000048ba9000]
    java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
    at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
    – locked (a java.lang.Object)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
    at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
    – locked (a com.sun.net.ssl.internal.ssl.AppInputStream)
    at oracle.ods.virtualization.engine.backend.jndi.OvdJndiInputStream.read(OvdJndiInputStream.java:59)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    – locked (a java.io.BufferedInputStream)
    at com.sun.jndi.ldap.Connection.run(Connection.java:808)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “Thread-226” daemon prio=10 tid=0x000000006114b000 nid=0x71e3 runnable [0x0000000048fad000]
    java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
    at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
    – locked (a java.lang.Object)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
    at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
    – locked (a com.sun.net.ssl.internal.ssl.AppInputStream)
    at oracle.ods.virtualization.engine.backend.jndi.OvdJndiInputStream.read(OvdJndiInputStream.java:59)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    – locked (a java.io.BufferedInputStream)
    at com.sun.jndi.ldap.Connection.run(Connection.java:808)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “Thread-225” daemon prio=10 tid=0x00000000616af000 nid=0x71e2 runnable [0x0000000048aa8000]
    java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
    at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
    – locked (a java.lang.Object)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
    at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
    – locked (a com.sun.net.ssl.internal.ssl.AppInputStream)
    at oracle.ods.virtualization.engine.backend.jndi.OvdJndiInputStream.read(OvdJndiInputStream.java:59)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    – locked (a java.io.BufferedInputStream)
    at com.sun.jndi.ldap.Connection.run(Connection.java:808)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “JPS UserRoleCacheCleaner” daemon prio=10 tid=0x0000000061777000 nid=0x71df waiting on condition [0x0000000046a88000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.security.jps.internal.idstore.util.UserRoleEntityCacheManager$PurgeThread.run(UserRoleEntityCacheManager.java:298)

    Locked ownable synchronizers:
    – None

    “Timer-9” daemon prio=10 tid=0x0000000060d42800 nid=0x71de in Object.wait() [0x0000000044866000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    – locked (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

    Locked ownable synchronizers:
    – None

    “Timer-8” daemon prio=10 tid=0x00002aaac49c2000 nid=0x71dc in Object.wait() [0x0000000045270000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    – locked (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

    Locked ownable synchronizers:
    – None

    “Timer-7” daemon prio=10 tid=0x0000000063b15800 nid=0x7197 in Object.wait() [0x0000000048dab000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    – locked (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

    Locked ownable synchronizers:
    – None

    “CacheManager” daemon prio=10 tid=0x0000000061453800 nid=0x7195 in Object.wait() [0x0000000048eac000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.lang.Object)
    at oracle.sysman.emSDK.svc.cache.CacheManager$Checker.run(CacheManager.java:790)
    – locked (a java.lang.Object)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “JobThreadPool:Long-System:Thread-199” daemon prio=10 tid=0x000000006054d800 nid=0x6cf3 in Object.wait() [0x0000000048caa000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “JobThreadPool:Long-System:Thread-198” daemon prio=10 tid=0x0000000060f64800 nid=0x6cea in Object.wait() [0x000000004809e000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “JobThreadPool:Long-System:Thread-197” daemon prio=10 tid=0x0000000060a06800 nid=0x6ce7 in Object.wait() [0x000000004506e000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “JobThreadPool:Long-System:Thread-173” daemon prio=10 tid=0x0000000061241800 nid=0x65a6 in Object.wait() [0x0000000046280000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “Timer-6” daemon prio=10 tid=0x000000005fc64000 nid=0x6016 in Object.wait() [0x00000000488a6000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    – locked (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

    Locked ownable synchronizers:
    – None

    “Thread-127” daemon prio=10 tid=0x00002aaabf09d000 nid=0x5fea waiting on condition [0x0000000046785000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.jdbc.pool.OracleImplicitConnectionCacheThread.run(OracleImplicitConnectionCacheThread.java:94)

    Locked ownable synchronizers:
    – None

    “Thread-126” daemon prio=10 tid=0x00002aaac42ad000 nid=0x5fc5 waiting on condition [0x0000000045b79000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at com.wles.PolicyCacheInvalidator.run(PolicyCacheInvalidator.java:269)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “SocketTimeout” daemon prio=10 tid=0x00002aaab67eb800 nid=0x5fbb waiting on condition [0x0000000046987000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at HTTPClient.SocketTimeout.run(StreamDemultiplexor.java:1204)

    Locked ownable synchronizers:
    – None

    “DynamicSSLListenThread[DefaultSecure]” daemon prio=10 tid=0x00002aaac401d000 nid=0x5f90 runnable [0x0000000046886000]
    java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
    – locked (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:462)
    at javax.net.ssl.impl.SSLServerSocketImpl.accept(Unknown Source)
    at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535)
    at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417)
    at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “[STANDBY] ExecuteThread: ‘5’ for queue: ‘weblogic.kernel.Default (self-tuning)'” daemon prio=10 tid=0x000000005fe21000 nid=0x5f8e in Object.wait() [0x0000000045e7c000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a weblogic.work.ExecuteThread)
    at java.lang.Object.wait(Object.java:485)
    at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:162)
    – locked (a weblogic.work.ExecuteThread)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:183)

    Locked ownable synchronizers:
    – None

    “weblogic.GCMonitor” daemon prio=10 tid=0x00002aaabf89e800 nid=0x5f8d in Object.wait() [0x000000004516f000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    – locked (a java.lang.ref.ReferenceQueue$Lock)
    at weblogic.platform.GCMonitorThread.waitForNotification(GCMonitorThread.java:88)
    at weblogic.platform.GCMonitorThread.run(GCMonitorThread.java:64)

    Locked ownable synchronizers:
    – None

    “Thread-122” prio=10 tid=0x0000000061285800 nid=0x5f8c waiting on condition [0x0000000045371000]
    java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    – parking to wait for (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)
    at weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:672)
    at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:721)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “WsMgmtWorkScheduler” daemon prio=10 tid=0x00002aaac4015800 nid=0x5f8b waiting on condition [0x0000000044a68000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.j2ee.ws.common.mgmt.util.WorkSchedulerImpl$1.run(WorkSchedulerImpl.java:37)

    Locked ownable synchronizers:
    – None

    “JobThreadPool:Wait:Thread-121” daemon prio=10 tid=0x000000006032f800 nid=0x5f8a in Object.wait() [0x0000000044e6c000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “JspTimeoutThread” daemon prio=10 tid=0x00002aaabe750000 nid=0x5f88 waiting on condition [0x00000000487a5000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.jsp.runtimev2.JspTimeoutThread.run(JspTimeoutThread.java:60)

    Locked ownable synchronizers:
    – None

    “MDSPollingThread-[em, jdbc/mds/sysman_mds]” daemon prio=10 tid=0x0000000060625000 nid=0x5f86 waiting on condition [0x00000000489a7000]
    java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    – parking to wait for (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
    at oracle.mds.internal.persistence.db.ChangePollingThread.run(ChangePollingThread.java:135)

    Locked ownable synchronizers:
    – None

    “Thread-119” daemon prio=10 tid=0x00002aaab7294800 nid=0x5f7f waiting on condition [0x0000000047f9d000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.jdbc.pool.OracleImplicitConnectionCacheThread.run(OracleImplicitConnectionCacheThread.java:94)

    Locked ownable synchronizers:
    – None

    “BadSQLEvaluation” daemon prio=10 tid=0x0000000061022000 nid=0x5f6e waiting on condition [0x00000000486a4000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.sysman.emdrep.dbjava.sqleval.BadSQLEvaluation.run(BadSQLEvaluation.java:164)

    Locked ownable synchronizers:
    – None

    “AgentWarnCleanupThread” daemon prio=10 tid=0x0000000060add800 nid=0x5f6b waiting on condition [0x00000000485a3000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.sysman.core.gccompliance.model.backend.ccc.thread.CCCThreadBase.run(CCCThreadBase.java:53)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “AnnotationCheckerThread” daemon prio=10 tid=0x000000005f14f800 nid=0x5f6a waiting on condition [0x00000000484a2000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.sysman.core.gccompliance.model.backend.ccc.thread.CCCThreadBase.run(CCCThreadBase.java:53)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “MetadataCheckThread” daemon prio=10 tid=0x000000005f726800 nid=0x5f63 waiting on condition [0x00000000483a1000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.sysman.core.gccompliance.model.backend.ccc.thread.CCCThreadBase.run(CCCThreadBase.java:53)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “PostloadThread” daemon prio=10 tid=0x000000005f28d800 nid=0x5f5e waiting on condition [0x00000000482a0000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.sysman.core.gccompliance.model.backend.ccc.thread.CCCThreadBase.run(CCCThreadBase.java:53)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “Thread-111” daemon prio=10 tid=0x000000005fa18800 nid=0x5f51 waiting on condition [0x000000004819f000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.jdbc.pool.OracleImplicitConnectionCacheThread.run(OracleImplicitConnectionCacheThread.java:94)

    Locked ownable synchronizers:
    – None

    “AdminMsgListener” daemon prio=10 tid=0x000000005fa15800 nid=0x5f4e runnable [0x0000000047e9c000]
    java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:718)
    at oracle.net.ns.Packet.receive(Packet.java:295)
    at oracle.net.ns.DataPacket.receive(DataPacket.java:106)
    at oracle.net.ano.CryptoDataPacket.receive(Unknown Source)
    at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:317)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:262)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:187)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:104)
    at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:126)
    at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:82)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1177)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1153)
    at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:312)
    at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:204)
    at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:540)
    at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:213)
    at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1075)
    at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1466)
    at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3752)
    at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3937)
    – locked (a oracle.jdbc.driver.T4CConnection)
    at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:9259)
    – locked (a oracle.jdbc.driver.T4CConnection)
    at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1535)
    at oracle.sysman.emSDK.core.util.jdbc.PreparedStatementWrapper.execute(PreparedStatementWrapper.java:243)
    at oracle.sysman.emSDK.core.util.jdbc.CallableStatementWrapper.execute(CallableStatementWrapper.java:145)
    at oracle.sysman.emdrep.adminmsg.AdminMsgListener.getAdminMsg(AdminMsgListener.java:513)
    at oracle.sysman.emdrep.adminmsg.AdminMsgListener.run(AdminMsgListener.java:223)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “OMSHeartbeatThread” daemon prio=10 tid=0x0000000060e3b800 nid=0x5f4d waiting on condition [0x0000000047896000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.sysman.emdrep.failover.OMSHeartbeatRecorder.run(OMSHeartbeatRecorder.java:542)

    Locked ownable synchronizers:
    – None

    “HealthMonitor” daemon prio=10 tid=0x000000005f493000 nid=0x5f4c in Object.wait() [0x0000000047b99000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.lang.Object)
    at oracle.sysman.util.healthMonitor.HealthMonitor.run(HealthMonitor.java:357)
    – locked (a java.lang.Object)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “PAFDaemon” daemon prio=10 tid=0x000000005f015000 nid=0x5f4b waiting on condition [0x0000000047d9b000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.sysman.emdrep.jobs.procedure.util.UnaryFunctionThread.sleepIfInactive(UnaryFunctionThread.java:421)
    at oracle.sysman.emdrep.jobs.procedure.util.UnaryFunctionThread.sleepIfNeeded(UnaryFunctionThread.java:479)
    at oracle.sysman.emdrep.jobs.procedure.util.UnaryFunctionThread.run(UnaryFunctionThread.java:548)

    Locked ownable synchronizers:
    – None

    “Timer-5” daemon prio=10 tid=0x00002aaab5801000 nid=0x5f4a in Object.wait() [0x0000000047c9a000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    – locked (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

    Locked ownable synchronizers:
    – None

    “ARU” daemon prio=10 tid=0x00002aaabeae5800 nid=0x5f43 in Object.wait() [0x0000000047a98000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    – locked (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

    Locked ownable synchronizers:
    – None

    “PATCH” daemon prio=10 tid=0x00002aaabe5b7000 nid=0x5f42 in Object.wait() [0x0000000047997000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    – locked (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

    Locked ownable synchronizers:
    – None

    “EndPointConnection CL-0 to CO at [192.168.1.178:23456]” daemon prio=10 tid=0x00002aaab6ce8000 nid=0x5f40 runnable [0x0000000047795000]
    java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    – locked (a java.io.BufferedInputStream)
    at oracle.ias.cache.groupv2.HttpHandler.read(HttpHandler.java:167)
    – locked (a oracle.ias.cache.groupv2.HttpHandler)
    at oracle.ias.cache.groupv2.EndPointConnection.receive(EndPointConnection.java:381)
    at oracle.ias.cache.groupv2.EndPointConnection.run(EndPointConnection.java:212)

    Locked ownable synchronizers:
    – None

    “Ack Processor” daemon prio=10 tid=0x00002aaabeae2800 nid=0x5f3f in Object.wait() [0x0000000047694000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.ias.cache.groupv2.AckProcessor)
    at java.lang.Object.wait(Object.java:485)
    at oracle.ias.cache.groupv2.AckProcessor.run(AckProcessor.java:53)
    – locked (a oracle.ias.cache.groupv2.AckProcessor)

    Locked ownable synchronizers:
    – None

    “CL-0 Packet Processor” daemon prio=10 tid=0x00002aaabe5ba000 nid=0x5f3e in Object.wait() [0x0000000047593000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.ias.cache.groupv2.GQueue)
    at oracle.ias.cache.groupv2.GQueue.qWait(GQueue.java:167)
    – locked (a oracle.ias.cache.groupv2.GQueue)
    at oracle.ias.cache.groupv2.GQueue.grabQs(GQueue.java:78)
    – locked (a oracle.ias.cache.groupv2.GQueue)
    at oracle.ias.cache.groupv2.Client.dequeueList(Client.java:1060)
    at oracle.ias.cache.groupv2.PacketProcessor.run(PacketProcessor.java:58)

    Locked ownable synchronizers:
    – None

    “EndPointConnection CO-0 to CL at [192.168.1.178:35195]” daemon prio=10 tid=0x00000000607c1000 nid=0x5f3d runnable [0x0000000047492000]
    java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    – locked (a java.io.BufferedInputStream)
    at oracle.ias.cache.groupv2.HttpHandler.read(HttpHandler.java:167)
    – locked (a oracle.ias.cache.groupv2.HttpHandler)
    at oracle.ias.cache.groupv2.EndPointConnection.receive(EndPointConnection.java:381)
    at oracle.ias.cache.groupv2.EndPointConnection.run(EndPointConnection.java:212)

    Locked ownable synchronizers:
    – None

    “Packet Sender” daemon prio=10 tid=0x00002aaabc361000 nid=0x5f3c in Object.wait() [0x0000000047391000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.ias.cache.groupv2.GQueue)
    at oracle.ias.cache.groupv2.GQueue.qWait(GQueue.java:167)
    – locked (a oracle.ias.cache.groupv2.GQueue)
    at oracle.ias.cache.groupv2.GQueue.grabQs(GQueue.java:78)
    – locked (a oracle.ias.cache.groupv2.GQueue)
    at oracle.ias.cache.groupv2.EndPointConnection.dequeueList(EndPointConnection.java:448)
    at oracle.ias.cache.groupv2.PacketSender.run(PacketSender.java:84)

    Locked ownable synchronizers:
    – None

    “Packet Sender” daemon prio=10 tid=0x00000000607c4000 nid=0x5f3b in Object.wait() [0x0000000047290000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.ias.cache.groupv2.GQueue)
    at oracle.ias.cache.groupv2.GQueue.qWait(GQueue.java:167)
    – locked (a oracle.ias.cache.groupv2.GQueue)
    at oracle.ias.cache.groupv2.GQueue.grabQs(GQueue.java:78)
    – locked (a oracle.ias.cache.groupv2.GQueue)
    at oracle.ias.cache.groupv2.EndPointConnection.dequeueList(EndPointConnection.java:448)
    at oracle.ias.cache.groupv2.PacketSender.run(PacketSender.java:84)

    Locked ownable synchronizers:
    – None

    “ViewMonitor of J23456:192.168.1.178:24001@nas.oracle.com” daemon prio=10 tid=0x00002aaab663e800 nid=0x5f3a waiting on condition [0x000000004718f000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.ias.cache.groupv2.ViewMonitor.run(ViewMonitor.java:93)

    Locked ownable synchronizers:
    – None

    “CO-0 Distributor at [192.168.1.178:23456]” daemon prio=10 tid=0x00002aaabf27f800 nid=0x5f39 runnable [0x000000004708e000]
    java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
    – locked (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:462)
    at java.net.ServerSocket.accept(ServerSocket.java:430)
    at oracle.ias.cache.groupv2.AccessCheck$20.run(AccessCheck.java:553)
    at java.security.AccessController.doPrivileged(Native Method)
    at oracle.ias.cache.groupv2.AccessCheck.accept(AccessCheck.java:550)
    at oracle.ias.cache.groupv2.Listener.run(Listener.java:142)

    Locked ownable synchronizers:
    – None

    “CO Packet Processor” daemon prio=10 tid=0x00002aaabc22b000 nid=0x5f38 in Object.wait() [0x0000000046f8d000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.ias.cache.groupv2.GQueue)
    at oracle.ias.cache.groupv2.GQueue.qWait(GQueue.java:167)
    – locked (a oracle.ias.cache.groupv2.GQueue)
    at oracle.ias.cache.groupv2.GQueue.grabQs(GQueue.java:78)
    – locked (a oracle.ias.cache.groupv2.GQueue)
    at oracle.ias.cache.groupv2.Distributor.dequeueList(Distributor.java:3104)
    at oracle.ias.cache.groupv2.PacketProcessor.run(PacketProcessor.java:58)

    Locked ownable synchronizers:
    – None

    “Timer-4” daemon prio=10 tid=0x00002aaab651c000 nid=0x5f37 in Object.wait() [0x0000000046c8a000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    – locked (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

    Locked ownable synchronizers:
    – None

    “HouseKeeping” daemon prio=10 tid=0x00002aaab62ba000 nid=0x5f34 in Object.wait() [0x0000000046d8b000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.util.TaskQueue)
    at java.lang.Object.wait(Object.java:485)
    at java.util.TimerThread.mainLoop(Timer.java:483)
    – locked (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

    Locked ownable synchronizers:
    – None

    “EmChartCacheWorker” daemon prio=10 tid=0x00002aaab551d800 nid=0x5f33 in Object.wait() [0x0000000046e8c000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.emSDK.chart.EmChartCacheWorker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.emSDK.chart.EmChartCacheWorker.run(EmChartCacheManager.java:468)
    – locked (a oracle.sysman.emSDK.chart.EmChartCacheWorker)

    Locked ownable synchronizers:
    – None

    “Thread-88” daemon prio=10 tid=0x000000005f47c800 nid=0x5f25 waiting on condition [0x0000000046b89000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.jdbc.pool.OracleImplicitConnectionCacheThread.run(OracleImplicitConnectionCacheThread.java:94)

    Locked ownable synchronizers:
    – None

    “EventWorker(Idle):Thread-83” daemon prio=10 tid=0x000000005f326800 nid=0x5ed4 in Object.wait() [0x0000000046684000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “EventWorker(Idle):Thread-82” daemon prio=10 tid=0x000000005ec3c000 nid=0x5ed3 in Object.wait() [0x0000000046583000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “EventWorker(Idle):Thread-81” daemon prio=10 tid=0x000000005ec3a800 nid=0x5ed2 in Object.wait() [0x0000000046482000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “EventWorker(Idle):Thread-80” daemon prio=10 tid=0x000000005f39e800 nid=0x5ed1 in Object.wait() [0x0000000046381000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “EventWorker(Idle):Thread-78” daemon prio=10 tid=0x0000000060a71800 nid=0x5ecf in Object.wait() [0x000000004617f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “EventWorker(Idle):Thread-77” daemon prio=10 tid=0x0000000060a70800 nid=0x5ece in Object.wait() [0x000000004607e000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “EventWorker(Idle):Thread-76” daemon prio=10 tid=0x0000000060e00000 nid=0x5ecd in Object.wait() [0x0000000045f7d000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “EventWorker(Idle):Thread-74” daemon prio=10 tid=0x000000005f079000 nid=0x5ecb in Object.wait() [0x0000000045d7b000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “EventWorker(Idle):Thread-73” daemon prio=10 tid=0x0000000060d80000 nid=0x5eca in Object.wait() [0x0000000045c7a000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “JobThreadPool:Long-System:Thread-71” daemon prio=10 tid=0x0000000060d81800 nid=0x5ec8 in Object.wait() [0x0000000045a78000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “JobThreadPool:Short-System:Thread-70” daemon prio=10 tid=0x0000000060d80800 nid=0x5ec7 in Object.wait() [0x0000000045977000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “JobThreadPool:Short-System:Thread-69” daemon prio=10 tid=0x0000000060571000 nid=0x5ec6 in Object.wait() [0x0000000045876000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “JobThreadPool:Long-Running:Thread-68” daemon prio=10 tid=0x000000006015a800 nid=0x5ec1 in Object.wait() [0x0000000045573000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “JobThreadPool:Long-Running:Thread-67” daemon prio=10 tid=0x000000006166e000 nid=0x5ec0 in Object.wait() [0x0000000040ce1000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “JobThreadPool:Long-Running:Thread-66” daemon prio=10 tid=0x000000006013b800 nid=0x5ebf in Object.wait() [0x0000000044967000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “JobThreadPool:Long-Running:Thread-65” daemon prio=10 tid=0x0000000061012000 nid=0x5ebe in Object.wait() [0x0000000045674000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “JobThreadPool:Short-Running:Thread-63” daemon prio=10 tid=0x000000005eb62800 nid=0x5ebc in Object.wait() [0x0000000045775000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “NotificationMgrThread” daemon prio=10 tid=0x00002aaab67c0800 nid=0x5eab runnable [0x0000000045471000]
    java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:718)
    at oracle.net.ns.Packet.receive(Packet.java:295)
    at oracle.net.ns.DataPacket.receive(DataPacket.java:106)
    at oracle.net.ano.CryptoDataPacket.receive(Unknown Source)
    at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:317)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:262)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:187)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:104)
    at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:126)
    at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:82)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1177)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1153)
    at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:312)
    at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:204)
    at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:540)
    at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:213)
    at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1075)
    at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1466)
    at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3752)
    at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3937)
    – locked (a oracle.jdbc.driver.T4CConnection)
    at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:9259)
    – locked (a oracle.jdbc.driver.T4CConnection)
    at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1535)
    at oracle.sysman.emSDK.core.util.jdbc.PreparedStatementWrapper.execute(PreparedStatementWrapper.java:243)
    at oracle.sysman.emSDK.core.util.jdbc.CallableStatementWrapper.execute(CallableStatementWrapper.java:145)
    at oracle.sysman.core.pbs.notification.NotificationMgr$CommonMaster.runCommon(NotificationMgr.java:543)
    at oracle.sysman.core.pbs.notification.NotificationMgr.run(NotificationMgr.java:2725)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “JobThreadPool:Wait:Thread-57” daemon prio=10 tid=0x000000005fe3f800 nid=0x5e88 in Object.wait() [0x0000000044f6d000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “JobThreadPool:Wait:Thread-55” daemon prio=10 tid=0x000000005e858800 nid=0x5e86 in Object.wait() [0x0000000044d6b000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “JobThreadPool:Wait:Thread-54” daemon prio=10 tid=0x000000005ff2f800 nid=0x5e85 in Object.wait() [0x0000000044c6a000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “JobThreadPool:Wait:Thread-53” daemon prio=10 tid=0x000000005f454800 nid=0x5e84 in Object.wait() [0x0000000044b69000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.util.threadPoolManager.Worker)
    at java.lang.Object.wait(Object.java:485)
    at oracle.sysman.util.threadPoolManager.Worker.getWork(Worker.java:127)
    – locked (a oracle.sysman.util.threadPoolManager.Worker)
    at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:263)

    Locked ownable synchronizers:
    – None

    “Job Dispatcher Thread” daemon prio=10 tid=0x000000005eb5a000 nid=0x5e7b waiting on condition [0x000000004405e000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.sysman.emdrep.jobs.Dispatcher.pauseThread(Dispatcher.java:947)
    at oracle.sysman.emdrep.jobs.Dispatcher.pause(Dispatcher.java:938)
    at oracle.sysman.emdrep.jobs.Dispatcher.run(Dispatcher.java:567)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “Thread-48” daemon prio=10 tid=0x00000000601a7000 nid=0x5e76 waiting on condition [0x0000000044765000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.jdbc.pool.OracleImplicitConnectionCacheThread.run(OracleImplicitConnectionCacheThread.java:94)

    Locked ownable synchronizers:
    – None

    “PolicyDispatcher” daemon prio=10 tid=0x00000000601a6800 nid=0x5e75 waiting on condition [0x0000000044462000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.sysman.core.ppc.engine.PolicyDispatcher.mainTask(PolicyDispatcher.java:262)
    at oracle.sysman.core.ppc.engine.PolicyDispatcher.run(PolicyDispatcher.java:161)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “ConnectorPoolManager” daemon prio=10 tid=0x000000006125b000 nid=0x5e74 in Object.wait() [0x0000000044361000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.lang.Object)
    at oracle.sysman.core.pbs.connector.ConnectorCoordinator.getQueueName(ConnectorCoordinator.java:741)
    – locked (a java.lang.Object)
    at oracle.sysman.core.pbs.connector.ConnectorPoolManager.run(ConnectorPoolManager.java:199)

    Locked ownable synchronizers:
    – None

    “Thread-43” daemon prio=10 tid=0x000000005f3a8800 nid=0x5e5a waiting on condition [0x0000000044664000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.jdbc.pool.OracleImplicitConnectionCacheThread.run(OracleImplicitConnectionCacheThread.java:94)

    Locked ownable synchronizers:
    – None

    “oracle.sysman.core.pbs.gcloader.ResourceManager” daemon prio=10 tid=0x000000005f3a7800 nid=0x5e59 waiting on condition [0x0000000044563000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.sysman.core.pbs.gcloader.ResourceManager.run(ResourceManager.java:1133)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “PingHeartBeatRecorder” daemon prio=10 tid=0x00000000605b8800 nid=0x5e49 in Object.wait() [0x0000000040be0000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.sysman.emdrep.ping.PingHeartBeatRecorder)
    at oracle.sysman.emdrep.ping.PingHeartBeatRecorder.waitRecordReady(PingHeartBeatRecorder.java:500)
    – locked (a oracle.sysman.emdrep.ping.PingHeartBeatRecorder)
    at oracle.sysman.emdrep.ping.PingHeartBeatRecorder.run(PingHeartBeatRecorder.java:533)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “ConnectorCoordinator” daemon prio=10 tid=0x000000005e8df000 nid=0x5e48 runnable [0x0000000044260000]
    java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:718)
    at oracle.net.ns.Packet.receive(Packet.java:295)
    at oracle.net.ns.DataPacket.receive(DataPacket.java:106)
    at oracle.net.ano.CryptoDataPacket.receive(Unknown Source)
    at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:317)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:262)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:187)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:104)
    at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:126)
    at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:82)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1177)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1153)
    at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:312)
    at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:204)
    at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:540)
    at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:213)
    at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1075)
    at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1466)
    at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3752)
    at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3937)
    – locked (a oracle.jdbc.driver.T4CConnection)
    at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:9259)
    – locked (a oracle.jdbc.driver.T4CConnection)
    at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1535)
    at oracle.sysman.emSDK.core.util.jdbc.PreparedStatementWrapper.execute(PreparedStatementWrapper.java:243)
    at oracle.sysman.emSDK.core.util.jdbc.CallableStatementWrapper.execute(CallableStatementWrapper.java:145)
    at oracle.sysman.core.pbs.connector.ConnectorCoordinator.listen(ConnectorCoordinator.java:699)
    at oracle.sysman.core.pbs.connector.ConnectorCoordinator.run(ConnectorCoordinator.java:495)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “Timer-2” daemon prio=10 tid=0x000000005f7fd800 nid=0x5e43 in Object.wait() [0x000000004415f000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    – locked (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

    Locked ownable synchronizers:
    – None

    “EventCoordinator” daemon prio=10 tid=0x000000005ef11800 nid=0x5e41 runnable [0x0000000043f5d000]
    java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:718)
    at oracle.net.ns.Packet.receive(Packet.java:295)
    at oracle.net.ns.DataPacket.receive(DataPacket.java:106)
    at oracle.net.ano.CryptoDataPacket.receive(Unknown Source)
    at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:317)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:262)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:187)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:104)
    at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:126)
    at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:82)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1177)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1153)
    at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:312)
    at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:204)
    at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:540)
    at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:213)
    at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1075)
    at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1466)
    at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3752)
    at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3937)
    – locked (a oracle.jdbc.driver.T4CConnection)
    at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:9259)
    – locked (a oracle.jdbc.driver.T4CConnection)
    at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1535)
    at oracle.sysman.emSDK.core.util.jdbc.PreparedStatementWrapper.execute(PreparedStatementWrapper.java:243)
    at oracle.sysman.emSDK.core.util.jdbc.CallableStatementWrapper.execute(CallableStatementWrapper.java:145)
    at oracle.sysman.core.pbs.events.EventCoordinator$CommonMaster.listen(EventCoordinator.java:1740)
    at oracle.sysman.core.pbs.events.EventCoordinator$CommonMaster.runCommon(EventCoordinator.java:1248)
    at oracle.sysman.core.pbs.events.EventCoordinator.run(EventCoordinator.java:757)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “PbsAdminMsgListener” daemon prio=10 tid=0x00000000600ce000 nid=0x5e40 runnable [0x0000000043e5b000]
    java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:718)
    at oracle.net.ns.Packet.receive(Packet.java:295)
    at oracle.net.ns.DataPacket.receive(DataPacket.java:106)
    at oracle.net.ano.CryptoDataPacket.receive(Unknown Source)
    at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:317)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:262)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:187)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:104)
    at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:126)
    at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:82)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1177)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1153)
    at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:312)
    at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:204)
    at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:540)
    at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:213)
    at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1075)
    at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1466)
    at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3752)
    at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3937)
    – locked (a oracle.jdbc.driver.T4CConnection)
    at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:9259)
    – locked (a oracle.jdbc.driver.T4CConnection)
    at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1535)
    at oracle.sysman.emSDK.core.util.jdbc.PreparedStatementWrapper.execute(PreparedStatementWrapper.java:243)
    at oracle.sysman.emSDK.core.util.jdbc.CallableStatementWrapper.execute(CallableStatementWrapper.java:145)
    at oracle.sysman.emdrep.adminmsg.AdminMsgListener.getAdminMsg(AdminMsgListener.java:513)
    at oracle.sysman.core.pbs.adminmsg.PbsAdminMsgListener.run(PbsAdminMsgListener.java:159)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “PbsOMSHeartbeatThread” daemon prio=10 tid=0x000000005ef14800 nid=0x5e3f waiting on condition [0x0000000043c5a000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.sysman.emdrep.failover.OMSHeartbeatRecorder.run(OMSHeartbeatRecorder.java:542)

    Locked ownable synchronizers:
    – None

    “PbsHealthMonitor” daemon prio=10 tid=0x000000005fce2000 nid=0x5e3e in Object.wait() [0x0000000043d5b000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.lang.Object)
    at oracle.sysman.util.healthMonitor.HealthMonitor.run(HealthMonitor.java:357)
    – locked (a java.lang.Object)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “oracle.dfw.common.DiagnosticsEventManager.NotifierThread” daemon prio=10 tid=0x000000005e944800 nid=0x5e39 in Object.wait() [0x0000000043755000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.util.LinkedList)
    at java.lang.Object.wait(Object.java:485)
    at oracle.dfw.common.DiagnosticsEventManager$NotifierThread.run(DiagnosticsEventManager.java:298)
    – locked (a java.util.LinkedList)
    at java.lang.Thread.run(Thread.java:662)

    Locked ownable synchronizers:
    – None

    “Thread-35” daemon prio=10 tid=0x0000000060331800 nid=0x5e30 waiting on condition [0x0000000043856000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.jdbc.pool.OracleImplicitConnectionCacheThread.run(OracleImplicitConnectionCacheThread.java:94)

    Locked ownable synchronizers:
    – None

    “JspTimeoutThread” daemon prio=10 tid=0x0000000061672800 nid=0x5e2f waiting on condition [0x0000000043a58000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.jsp.runtimev2.JspTimeoutThread.run(JspTimeoutThread.java:60)

    Locked ownable synchronizers:
    – None

    “Thread-33” daemon prio=10 tid=0x0000000061a3b000 nid=0x5e2e waiting on condition [0x0000000043957000]
    java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    – parking to wait for (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    at oracle.mds.internal.notifications.EventDispatcherThread.run(EventDispatcherThread.java:57)

    Locked ownable synchronizers:
    – None

    “MDSPollingThread-[APM, jdbc/mds/mds-ApplicationMDSDBDS]” daemon prio=10 tid=0x00002aaab63da000 nid=0x5e2d waiting on condition [0x0000000043b59000]
    java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    – parking to wait for (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
    at oracle.mds.internal.persistence.db.ChangePollingThread.run(ChangePollingThread.java:135)

    Locked ownable synchronizers:
    – None

    “MDSPollingThread-[owsm, jdbc/mds/owsm]” daemon prio=10 tid=0x00002aaab5fa4000 nid=0x5e27 waiting on condition [0x0000000040f0b000]
    java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    – parking to wait for (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
    at oracle.mds.internal.persistence.db.ChangePollingThread.run(ChangePollingThread.java:135)

    Locked ownable synchronizers:
    – None

    “CacheCleaner” daemon prio=10 tid=0x00002aaab5557000 nid=0x5e25 in Object.wait() [0x0000000043654000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.ias.cache.TimeQ)
    at oracle.ias.cache.TimeQ.waitForWork(TimeQ.java:178)
    – locked (a oracle.ias.cache.TimeQ)
    at oracle.ias.cache.CacheCleaner.run(CacheCleaner.java:116)

    Locked ownable synchronizers:
    – None

    “Worker-2” daemon prio=10 tid=0x00002aaab5f93800 nid=0x5e24 in Object.wait() [0x0000000043553000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a oracle.ias.cache.TaskQ)
    at oracle.ias.cache.TaskQ.waitForWork(TaskQ.java:255)
    – locked (a oracle.ias.cache.TaskQ)
    at oracle.ias.cache.TaskQ.getTask(TaskQ.java:138)
    – locked (a oracle.ias.cache.TaskQ)
    at oracle.ias.cache.WorkerThread.run(ThreadPool.java:303)

    Locked ownable synchronizers:
    – None

    “Worker-1” daemon prio=10 tid=0x00002aaab7f31000 nid=0x5e23 in Object.wait() [0x0000000043452000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Ob

Trackbacks

  1. […] Script:收集Enterprise Manager Grid Control/Agent/Database Control诊断信息 Script:收集Exadata诊断信息 Script:收集RAC诊断信息 Script:收集自动SGA内存管理ASMM诊断信息 Script:Collect vip resource Diagnostic Information 11g新特性:hangdiag.sql实例hang诊断脚本 Script:verify Oracle Object timestamp discrepancy Script:SQL调优健康检查脚本 Script:列出本会话的细节信息 利用rowid分块实现非分区表的并行update与delete Script:计算Oracle Streams进程所占用的内存大小 利用RMAN检测数据库坏块的脚本 Script:利用外部表实现SQL查询Oracle告警日志Alert.log Script: 收集RAC DRM 诊断信息 Script:10g中不用EM显示Active Session Count by Wait Class Script:数据库最近的性能度量 Script:收集数据库中用户的角色和表空间等信息 Script:收集介质恢复诊断信息 Script:收集Flashback Database Log诊断信息 Script:列出Oracle每小时的redo重做日志产生量 Script:收集11g Oracle实例IO性能信息 Script:检查数据库当前是否有备份操作在执行中 Script:List Schema/Table Constraints Script:RAC Failover检验脚本loop.sh Script:Diagnostic Resource Manager Script:List Grid Control Jobs Script:GridControl Repository Health Check Script:诊断Scheduler信息 Script:优化crs_stat命令的输出 Script:Diagnostic Oracle Locks Script:列出用户表空间的定额 Backup Script:Expdp Schema to ASM Storage Script:Speed Up Large Index Create or Rebuild Script:列出失效索引或索引分区 Script:列出数据库中5%以上链式行的表 Script:列出没有主键或唯一索引的表 Script:收集ASM诊断信息 Script:收集Oracle备份恢复信息 监控一个大事务的回滚 Script to Collect DB Upgrade/Migrate Diagnostic Information (dbupgdiag.sql) Script:partition table into rowid extent chunks Script:Oracle EBS数据库初始化参数健康检查脚本 Script:Monitoring Memory and Swap Usage to Avoid A Solaris Hang SQL脚本:监控当前重做日志文件使用情况 Streams Health Check on 10g Release 2 从视图查询表分区的相关信息 Script To Monitor RDBMS Session UGA and PGA Current And Maximum Usage Over Time Script:收集RAC性能诊断信息 Script:收集UNDO诊断信息 Script:列出数据库中子表上没有对应索引的外键 Script: Listing Memory Used By All Sessions Collecting Diagnostic Data for OCFS2 Issues Script to Identify Objects and Amount of Blocks in the Buffer Pools – Default, Keep, Recycle, nK Cache Script:Generate A DDL Script For A Table SCRIPT TO CHECK FOR FOREIGN KEY LOCKING ISSUES 如何找出Oracle中需要或值得重建的索引 Script:Diagnostic ORA-01000 maximum open cursors exceeded ORA-4030 PGA Usage Diagnostic Script Script:Tune Very Large Hash Join Script to Collect Log File Sync Diagnostic Information (lfsdiag.sql) Script:List Buffer Cache Details Script:List NLS Parameters and Timezone Script:List SORT ACTIVITY Script:List OBJECT DEPENDENT Script:Logfile Switch Frequency Map Script:Tablespace Report Script:收集数据库安全风险评估信息 脚本:格式化的V$SQL_SHARED_CURSOR报告 脚本:监控并行进程状态 脚本:监控数据库中的活跃用户及其运行的SQL 脚本:监控临时表空间使用率 Script to show Active Distributed Transactions Gather DBMS_STATS Default parameter Script:Datafile Report Script to Collect Data Guard Diagnostic Information Script:To Report Information on Indexes ORA-4031 Common Analysis/Diagnostic Scripts Script:when transaction will finish rollback Script: Computing Table Size Script to Detect Tablespace Fragmentation “hcheck.sql” script to check for known problems in Oracle8i, Oracle9i, Oracle10g and Oracle 11g Script to Prevent Excessive Spill of Message From the Streams Buffer Queue To Disk Oracle Systemstate dump analytic tool: ASS.AWK V1.09 SCRIPT TO GENERATE SQL*LOADER CONTROL FILE […]

Comment

*

沪ICP备14014813号-2

沪公网安备 31010802001379号