一套Sparc Solaris上的11.1.0.7系统，在创建索引时频繁出现ORA-00600: internal error code, arguments: [15801], [1], [], [], [], [], [], [], [], [], [], []内部错误，日志信息如下:
Tue Aug 17 17:34:21 2010 WARNING: Oracle executable binary mismatch detected. Binary of new process does not match binary which started instance issue alter system set "_disable_image_check" = true to disable these messages Tue Aug 17 17:34:21 2010 Errors in file /u04/app/oracle/diag/rdbms/orahcmu/ORAHCMU/trace/ORAHCMU_p023_22262.trc (incident=12505): ORA-00600: internal error code, arguments: [15801], [1], [], [], [], [], [], [], [], [], [], [] Incident details in: /u04/app/oracle/diag/rdbms/orahcmu/ORAHCMU/incident/incdir_12505/ORAHCMU_p023_22262_i12505.trc Tue Aug 17 17:34:21 2010 Errors in file /u04/app/oracle/diag/rdbms/orahcmu/ORAHCMU/trace/ORAHCMU_p021_22258.trc (incident=12489): ORA-00600: internal error code, arguments: [15801], [1], [], [], [], [], [], [], [], [], [], [] Incident details in: /u04/app/oracle/diag/rdbms/orahcmu/ORAHCMU/incident/incdir_12489/ORAHCMU_p021_22258_i12489.trc Errors in file /u04/app/oracle/diag/rdbms/orahcmu/ORAHCMU/trace/ORAHCMU_p015_9328.trc (incident=19909): ORA-00600: internal error code, arguments: [15801], [1], [], [], [], [], [], [], [], [], [], [] Errors in file /u04/app/oracle/diag/rdbms/orahcmu/ORAHCMU/trace/ORAHCMU_p043_9388.trc (incident=20133): ORA-00600: internal error code, arguments: [15801], [1], [], [], [], [], [], [], [], [], [], [] Mon Aug 23 14:43:42 2010 Errors in file /u04/app/oracle/diag/rdbms/orahcmu/ORAHCMU/trace/ORAHCMU_p087_9668.trc (incident=20485): ORA-00600: internal error code, arguments: [15801], [1], [], [], [], [], [], [], [], [], [], [] Mon Aug 23 14:43:42 2010 Errors in file /u04/app/oracle/diag/rdbms/orahcmu/ORAHCMU/trace/ORAHCMU_p012_9322.trc (incident=19885): ORA-00600: internal error code, arguments: [15801], [1], [], [], [], [], [], [], [], [], [], [] Incident details in: /u04/app/oracle/diag/rdbms/orahcmu/ORAHCMU/incident/incdir_19789/ORAHCMU_ora_8602_i19789.trc Mon Aug 23 14:43:43 2010 WARNING: Oracle executable binary mismatch detected. Binary of new process does not match binary which started instance issue alter system set "_disable_image_check" = true to disable these messages Dump continued from file: /u04/app/oracle/diag/rdbms/orahcmu/ORAHCMU/trace/ORAHCMU_ora_8602.trc ORA-00600: internal error code, arguments: [15801], [1], [], [], [], [], [], [], [], [], [], [] *** 2010-08-23 14:43:42.974 ----- Current SQL Statement for this session (sql_id=00abhfx460qm9) ----- CREATE UNIQUE iNDEX PS_HM_BEN_GP_STG ON PS_HM_BEN_GP_STG (CAL_ID, GP_PAYGROUP, EMPLID, EMPL_RCD, HM_INCURRED_BY, HM_SUM_ASSURED) TABLESPACE PSINDEX STORAGE (INITIAL 40000 NEXT 100000 MAXEXTENTS UNLIMITED PCTINCREASE 0) PCTFREE 10 PARALLEL NOLOGGING ----- Call Stack Trace ----- ksedst1 ksedst dbkedDefDump dbgexPhaseII dbgexProcessError dbgePostErrorKGE kgeade kgerem kxfpProcessError kxfpqidqr kxfpqdqr kxfxgs kxfxcp qerpxSendParse kxfpValidateSlaveGroup kxfpgsg kxfrAllocSlaves kxfrialo kxfralo qerpx_rowsrc_start qerpxStart kdicrws kdicdrv opiexe opiosq0 kpooprx kpoal8 opiodr ttcpip opitsk opiino opiodr opidrv sou2o main SO: 0x3bf0bbf20, type: 4, owner: 0x3bf5452d0, flag: INIT/-/-/0x00 if: 0x3 c: 0x3 proc=0x3bf5452d0, name=session, file=ksu.h LINE:10719 ID:, pg=0 (session) sid: 217 ser: 767 trans: 0x3bc0660f8, creator: 0x3bf5452d0 flags: (0x8000041) USR/- flags_idl: (0x1) BSY/-/-/-/-/- flags2: (0x44008) DDLT1/- DID: , short-term DID: txn branch: 0x0 oct: 9, prv: 0, sql: 0x3b5d14510, psql: 0x3b6d59820, user: 31/SYSADM ksuxds FALSE at location: 0 service name: ORAHCMU client details: O/S info: user: Administrator, term: UJWALTPVM, ospid: 304:2892 machine: WORKGROUP\UJWALTPVM program: pside.exe client info: ujwal,Administrator,UJWALTPVM,,pside.exe, application name: pside.exe, hash value=2824484291 Current Wait Stack: Not in wait; last wait ended 2.475286 sec ago Wait State: auto_close=0 flags=0x21 boundary=0x0/-1 Session Wait History: 0: waited for 'lient' =c8, =1, =0 wait_id=10483 seq_num=10484 snap_id=1 wait times: snap=0.168502 sec, exc=0.168502 sec, total=0.168502 sec wait times: max=2.000000 sec wait counts: calls=1 os=1 occurred after 0.000903 sec of elapsed time 1: waited for ' waiting for ruleset' =10010063, =1, =0 wait_id=10482 seq_num=10483 snap_id=1 wait times: snap=0.008580 sec, exc=0.008580 sec, total=0.008580 sec wait times: max=2.000000 sec wait counts: calls=1 os=1 occurred after 0.000731 sec of elapsed time 2: waited for ' waiting for ruleset' =1001004f, =4, =0 wait_id=10481 seq_num=10482 snap_id=1 wait times: snap=0.000132 sec, exc=0.000132 sec, total=0.000132 sec wait times: max=2.000000 sec wait counts: calls=1 os=1 occurred after 0.000074 sec of elapsed time 3: waited for ' waiting for ruleset' =1001004f, =3, =0 wait_id=10480 seq_num=10481 snap_id=1 wait times: snap=0.000002 sec, exc=0.000002 sec, total=0.000002 sec wait times: max=2.000000 sec wait counts: calls=1 os=1 occurred after 0.000065 sec of elapsed time ----- Session Cursor Dump ----- Current cursor: 1, pgadep=0 Open cursors(pls, sys, hwm, max): 3(0, 2, 64, 300) NULL=1 SYNTAX=0 PARSE=0 BOUND=1 FETCH=0 ROW=1 Cached frame pages(total, free): 4k(14, 14), 8k(1, 1), 16k(1, 1), 32k(0, 0) ----- Current Cursor ----- ----- Plan Table ----- ============ Plan Table ============ ----------------------------------------------------+-----------------------------------+-------------------------+ | Id | Operation | Name | Rows | Bytes | Cost | Time | TQ |IN-OUT|PQ Distrib | ----------------------------------------------------+-----------------------------------+-------------------------+ | 0 | CREATE INDEX STATEMENT | | | | 2 | | | | | | 1 | PX COORDINATOR | | | | | | | | | | 2 | PX SEND QC (ORDER) | :TQ10001 | 82 | 4510 | | |:Q1001| P->S |QC (ORDER) | | 3 | INDEX BUILD UNIQUE | PS_HM_BEN_GP_STG| | | | |:Q1001| PCWP | | | 4 | SORT CREATE INDEX | | 82 | 4510 | | |:Q1001| PCWP | | | 5 | PX RECEIVE | | 82 | 4510 | 2 | 00:00:01 |:Q1001| PCWP | | | 6 | PX SEND RANGE | :TQ10000 | 82 | 4510 | 2 | 00:00:01 |:Q1000| P->P |RANGE | | 7 | PX BLOCK ITERATOR | | 82 | 4510 | 2 | 00:00:01 |:Q1000| PCWC | | | 8 | TABLE ACCESS FULL | PS_HM_BEN_GP_STG| 82 | 4510 | 2 | 00:00:01 |:Q1000| PCWP | | ----------------------------------------------------+-----------------------------------+-------------------------+ ---------------------------------------- Cursor#1(0xffffffff7ce31928) state=BOUND curiob=0xffffffff7ce57d28 curflg=4c fl2=0 par=0x0 ses=0x3bf0bbf20 ----- Dump Cursor sql_id=00abhfx460qm9 xsc=0xffffffff7ce57d28 cur=0xffffffff7ce31928 ----- Dump Parent Cursor sql_id=00abhfx460qm9 phd=0x3b5d14510 plk=0x3b0bb3318 sqltxt(0x3b5d14510)=CREATE UNIQUE iNDEX PS_HM_BEN_GP_STG ON PS_HM_BEN_GP_STG (CAL_ID, GP_PAYGROUP, EMPLID, EMPL_RCD, HM_INCURRED_BY, HM_SUM_ASSURED) TABLESPACE PSINDEX STORAGE (INITIAL 40000 NEXT 100000 MAXEXTENTS UNLIMITED PCTINCREASE 0) PCTFREE 10 PARALLEL NOLOGGING hash=616eaa631fc21f4c0029707748605a69 parent=0x3ae539590 maxchild=01 plk=0x3b0bb3318 ppn=n cursor instantiation=0xffffffff7ce57d28 used=1282545779 exec_id=16777216 exec=1 child#0(0x3b5d05e10) pcs=0x3b678c128 clk=0x3b7e200d0 ci=0x3b5b204c8 pn=0x39955d2b8 ctx=0x3b86ee988 kgsccflg=0 llk[0xffffffff7ce57d30,0xffffffff7ce57d30] idx=0 xscflg=c0102276 fl2=c000400 fl3=2202008 fl4=100 Frames pfr 0xffffffff7ce67098 siz=85976 efr 0xffffffff7ce66fb8 siz=85960 Cursor frame dump enxt: 7.0x00000168 enxt: 6.0x00008000 enxt: 5.0x00008000 enxt: 4.0x00003978 enxt: 3.0x00000490 enxt: 2.0x000000b8 enxt: 1.0x00000fa0 pnxt: 1.0x00000010 kxscphp=0xffffffff7dd80a18 siz=984 inu=312 nps=312 kxscwhp=0xffffffff7ddd2cc8 siz=8136 inu=6264 nps=3968 kxscefhp=0xffffffff7ce51468 siz=88456 inu=86128 nps=86128 FileName ---------------- ORAHCMU_ora_8602.trc FileComment ---------------------- Oracle Support - August 27, 2010 6:13:39 PM GMT+08:00 [ODM Data Collection] Name -------- === ODM Data Collection === === ODM Data Collection === Trace file /u04/app/oracle/diag/rdbms/orahcmu/ORAHCMU/trace/ORAHCMU_p012_9322.trc *** 2010-08-23 14:43:00.472 WARNING: Oracle executable binary mismatch detected. Binary of new process does not match binary which started instance issue alter system set "_disable_image_check" = true to disable these messages startup image information iid info sz=245752512 inode=65458 ts=0x4c6df668 current process image information iid info sz=245750720 inode=65427 ts=0x4c7204b0 set _disable_image_check = TRUE to disable this check qksceLinearToCe error *** 2010-08-23 14:43:42.974 *** SESSION ID:(220.111) 2010-08-23 14:43:42.974 *** CLIENT ID:(ujwal) 2010-08-23 14:43:42.974 *** SERVICE NAME:(ORAHCMU) 2010-08-23 14:43:42.974 DDE: Problem Key 'ORA 600 [15801]' was flood controlled (0x6) (incident: 19885) ORA-00600: internal error code, arguments: [15801], [1], [], [], [], [], [], [], [], [], [], [] kxfxdss KXFXSLAVESTATE dump [0, 0] (pgakid: 0 oercnt: 0 oerrcd: -2224892588) kxfxdss no current cursor context. kxfxdss no cursors.
关于binary no match的问题已知是由于在实例启动情况下relink导致的；这个case提交了SR，metalink认为ORA-600 15801一般由QC与服务子进程通信问题引起:
The ORA-600 15801 is reporting a communication problem between QC and slaves related with messages sent/received.
Alert log reports several of the following error on the ASM instance:
ORA-600: internal error code, arguments: [15801], [1], [], [], [], [], [],
[]
last wait was for 'eq: Msg Fragment'
DIAGNOSTIC ANALYSIS:
--------------------
There were also several of the following message in the alert log:
WARNING: Oracle executable binary mismatch detected.
Binary of new process does not match binary which started instance
issue alter system set "_disable_image_check" = true to disable these
messages
So, I asked the customer to set the "_disable_image_check" = true
This had no impact on the ora-600 errors as expected.
ORA-600 [15801] is signalled when a message overflow occurs between PQ
processes.
WORKAROUND:
-----------
none
RELATED BUGS:
-------------
none
REPRODUCIBILITY:
----------------
intermittent but frequently - occurs at all different times of the day.
STACK TRACE:
------------
*** ID:(29.2904) 2006-07-05 15:50:57.972
qksceLinearToCe error
*** 15:50:58.233
ksedmp: internal or fatal error
ORA-600: internal error code, arguments: [15801], [1], [], [], [], [], [],
[]
----- Call Stack Trace -----
kxfxGeter qks3tttdefReceive kxfxsui kxfxsp kxfxmai kxfprdp
SO: 0x67977018, type: 4, owner: 0x6793f208, flag: INIT/-/-/0x00
(session) sid: 29 trans: (nil), creator: 0x6793f208, flag: (c0000041)
USR/- BSY/-/-/-/-/-
DID: 0000-0012-0000FADB, short-term DID: 0000-0000-00000000
txn branch: (nil)
oct: 3, prv: 0, sql: (nil), psql: (nil), user: 0/SYS
O/S info: user: oracle, term: , ospid: 4558, machine:
last wait for 'eq: Msg Fragment' blocking sess=0x(nil) seq=2
wait_time=4441 seconds since wait started=3
ct path write=1002ffff, ct path write temp=2, Network=0
Dumping Session Wait History
for 'eq: Msg Fragment' count=1 wait_time=4441
ct path write=1002ffff, ct path write temp=2, Network=0
for 'eq: Msg Fragment' count=1 wait_time=31
ct path write=1002ffff, ct path write temp=1, Network=0
temporary object counter: 0
最后这个case通过设置10235和10501事件后错误不再产生了:
event = "10235 trace name context forever, level 2" 10235, 00000, "check memory manager internal structures" event = "10501 trace name context forever, level 1" 10501, 00000, "periodically check selected heap" // *Cause: // *Action: // Level: 0x01 PGA // 0x02 SGA // 0x04 UGA // 0x08 current call // 0x10 user call // 0x20 large allocation pool
Comment